Archive for the ‘google’ Category

Monday, May 13th, 2013

Robots.txt and Invisible Characters – How One Hidden Character Could Cause SEO Problems

How syntax errors in robots.txt can cause SEO problems.

If you’ve read some of my blog posts in the past, then you know I perform a lot of SEO technical audits. As one of the checks during SEO audits, I always analyze a client’s robots.txt file to ensure it’s not blocking important directories or files. If you’re not familiar with robots.txt, it’s a text file that sits in the root directory of your website and should be used to inform the search engine bots which directories or files they should not crawl. You can also add autodiscovery for your xml sitemaps (which is a smart directive to add to a robots.txt file).

Anyway, I came across an interesting situation recently that I wanted to share. My hope is that this post can help some companies avoid a potentially serious SEO issue that was not readily apparent. Actually, the problem could not be detected by the naked eye. And when a problem impacts your robots.txt file, the bots won’t follow your instructions. And when the bots don’t follow instructions, they can potentially be unleashed into content that should never get crawled. Let’s explore this situation in greater detail.

A sample robots.txt file:

Sample Robots.txt File

Technical SEO – Cloaked Danger in a Robots.txt File
During my first check of the robots.txt file, everything looked fine. There were a number of directories being blocked for all search engines. Autodiscovery was added, which was great. All looked good. Then I checked Google Webmaster Tools to perform some manual checks on various files and directories (based on Google’s “Blocked URLs” functionality). Unfortunately, there were a number of errors showing within the analysis section.

The first error message started with the User-agent line (the first line in the file). Googlebot was choking on that line for some reason, but it looked completely fine. And as you can guess, none of the directives listed in the file were being adhered to. This meant that potentially thousands of files would be crawled that shouldn’t be crawled, and all because of a problem that was hiding below the surface… literally.

Blocked URLs reporting in Google Webmaster Tools:

Blocked URLs in Google Webmaster Tools

Word Processors and Hidden Characters
So I started checking several robots.txt tools to see what they would return. Again, the file looked completely fine to me. The first few checks returned errors, but wouldn’t explain exactly what was wrong. And then I came across one that revealed more information. The tool revealed an extra character (hidden character) at the beginning of the robots.txt file. This hidden character was throwing off the format of the file, and the bots were choking on it. And based on the robots syntax being thrown off, the bots wouldn’t follow the instructions. Not good.

Invisible Character in Robots.txt

I immediately sent this off to my client and their dev team tracked down the hidden character, and created a new robots.txt file. The new file was uploaded pretty quickly (within a few hours). And all checks are fine now. The bots are also adhering to the directives included in robots.txt.

The SEO Problems This Scenario Raises
I think this simple example underscores the fact that there’s not a lot of room for error with technical SEO… it must be precise. In this case, one hidden character in a robots.txt file unleashed the bots on a lot of content that should never be crawled. Sure, there are other mechanisms to make sure content doesn’t get indexed, like the proper use of the meta robots tag, but that’s for another post. For my client, a robots.txt file was created, it looked completely fine, but one character was off (and it was hidden). And that one character forced the bots to choke on the file.

How To Avoid Robots.txt Formatting Issues
I think one person at my client’s company summed up this situation perfectly when she said, “it seems you have little room for error, SEO seems so delicate”. Yes, she’s right (with technical SEO). Below, I’m going to list some simple things you can do to avoid this scenario. If you follow these steps, you could avoid faulty robots.txt files that seem accurate to the naked eye.

1. Text Editors
Always use a text editor when creating your robots.txt file. Don’t use a word processing application like Microsoft Word. A text editor is meant to create raw text files, and it won’t throw extra characters into your file by accident.

2. Double and Triple Check Your robots.txt Directives
Make sure each directive does exactly what you think it will do. If you aren’t 100% sure you know, then ask for help. Don’t upload a robots.txt file that could potentially block a bunch of important content (or vice versa).

3. Test Your robots.txt File in Google Webmaster Tools and Via Third Party Tools
Make sure the syntax of your robots.txt file is correct and that it’s blocking the directories and files you want it to. Note, Google Webmaster Tools enables you to copy and paste a new robots file into a form and test it out. I highly recommend you do this BEFORE uploading a new file to your site.

4. Monitor Google Webmaster Tools “Blocked URLs” Reporting
The blocked urls functionality will reveal problems associated with your robots.txt file under the “analysis” section. Remember, this is where I picked up the problem covered in this post.

Extra Characters in Robots.txt – Cloaked in Danger
There you have it. One hidden character bombed a robots.txt file. The problem was hidden to the naked eye, but the bots were choking on it. And depending on your specific site, that one character could have led to thousands of pages getting crawled that shouldn’t be. I hope this post helped you understand that your robots.txt format and syntax are extremely important, that you should double and triple check your file, and that you can test and monitor that file over time. If the wrong file is uploaded to your website, bad things can happen. Avoid this scenario.

Posted by Glenn Gabe @ 8:02 am in bing, google, seo, tools

No Comments »

Sunday, March 24th, 2013

Facebook Graph Search Update: Facebook Now Passing Keywords to Destination Websites, Can Track in Google Analytics

Facebook Graph Search launched in January, and I was fortunate to have early access. Upon gaining access, I began heavily testing Facebook’s new social search capabilities. That research led to my first post covering my initial thoughts about Graph Search, including its impact on local search, privacy, reputation management, etc. You should check out that post out after reading through this one.

Graph Search and Not Provided
One (unfortunate) point I covered in my post was the apparent use of “not provided” in Graph Search. If you’re not familiar with “not provided”, last year Google started encrypting searches for users that were logged into a Google account. As a result, the keywords that users searched for would not be passed in the referrer when they visited your website. Essentially, webmasters started seeing “not provided” show up in their analytics reporting versus the organic keywords that led to the site. Needless to say, this was a huge problem and the “not provided” numbers have been growing ever since. For some websites, “not provided” accounts for over 60% of all organic searches.

Now back to Facebook Graph Search and how it handles searches. When Graph Search launched, I quickly realized that Graph Search falls back to Bing results when it can’t answer your query. You can read my post about Graph Search to learn about the autocomplete funnel that you get forced down. So, if you aren’t searching for photos, people, places, or interests, then Facebook will display the “web results” for the query (which is basically the Bing results).

During my testing, I quickly checked the referrer when clicking from the fallback results to websites and noticed that Facebook was not passing the keyword in the referrer! It was basically Facebook’s version of “not provided”. When you checked Google Analytics, the visit looked like a typical referring source (a “Social” visit from Facebook). You couldn’t see the keyword that triggered the visit. Needless to say, I thought Facebook was making a big mistake. If Facebook passed the keywords along, business owners, advertisers, SEO’s, etc. would love them for doing so… We could all start to track the impact of Graph Search (at least any searches that fall back to Bing). I guess we were out of luck… or were we?

SES NY Session Research and the “Referrer” Moment
SES NY is this upcoming week, and my session covers Facebook Graph Search (based on the research I mentioned earlier). While I was finalizing my presentation over the past week, I noticed something very interesting. I started a slide on “not provided”, and then double-checked some searches leading from Facebook to external websites. What I found surprised me!

I noticed that the keyword was now being passed along in the referrer! Below is an example of a new referrer. You can clearly see the q={keyword} being passed now.

That’s awesome, but now we need to track the keywords. Read on.

Google Analytics and Graph Search
As you can guess, I jumped into Google Analytics to see how this was being picked up. Since Facebook isn’t an official search engine in GA, it was still showing up as a referring site (without the keyword showing up). But, since the q= querystring parameter was being passed in the referrer, I knew I could surface those keywords via advanced filters. So, I quickly set up a new profile and added a filter that would capture graph searches from Facebook. And it works. I explain how to quickly set this up below.

How to Track Graph Searches in GA
I’m not going to explain the inner workings of advanced filters in this post (it could easily be a series of posts). Instead, I’ll show you how to quickly set up an advanced filter that will capture graph search keywords, and then report them in Google Analytics. Note, you can tackle this several ways, and this is just one solution. Feel free to tailor your own GA setup as you see fit.

1. Launch Google Analytics and Access Your Website Profiles
You can access your profiles by clicking the “Admin” button in the upper right-hand corner of the GA interface. The first tab should list all profiles for the website at hand (for the “property” in GA).

2. Create a New Profile
Click the “New Profile” button and give it a name. Then choose your reporting time zone. Then click the “Create Profile” button.

3. Add an Advanced Filter
Once you create your profile, you should click the “Filters” tab so you can add an advanced filter.

4. Adding a New Filter
Click the “New Filter” button, and then make sure the settings match what is list below. We will name the filter, create a custom filter, select advanced, and then enter patterns to match in order to capture graph search keywords (and then report them in GA).

5. Save the Filter
Click “Save” at the bottom of the form and you should see your new filter listed for your profile.

6. Check Your Reporting
Now you need to wait for graph searches to come in. Note, this filter will just pick up graph searches that fall back to Bing and that lead to your site. If you have access to Graph Search, then you can quickly test it out by searching Facebook for a query that your site ranks for in Bing organic. Then click through the Facebook Graph Search results and check your reporting later on (the standard GA reporting isn’t real-time, so the reports will lag slightly).

To find the new data, you can click the “Sources” Tab, and then click “All Traffic”. Then find “facebook.com” and click through. Then you can add a secondary dimension for “User Defined Value” which will contain your graph search keywords. When setting the secondary dimension in your reporting, click “Content” and then select “User Defined Value”. Note, the new advanced filter will capture all q={keyword} combinations. That’s why you need to drill into “facebook.com” in the sources to isolate Graph Search results.

Your reporting will look something like this:

Once you start picking up graph searches, you can tie those searches to site activity, performance, conversion, etc. In addition, we can start to see how many graph searches are being conducted that drive users off of Facebook.com (meaning those searches can’t be answered yet by Facebook Graph Search).

Summary – Capture Graph Search Keywords
Again, I’m glad to see that Facebook is passing keywords along in the referrer! For a while, I thought we were going to be stuck with another “not provided” situation. Although Facebook Graph Search has only been rolled out to a few hundred thousand users, it’s only a matter of time before the 1 billion+ active user base gains access. And that will be across mobile devices too. My recommendation is to set up this advanced filter today and start tracking graph searches. You never know what you are going to find. :)

Last, but not least, if you are attending SES NY this week, and are interested in learning more about Facebook Graph Search, you should definitely check out my session. It’s at 2:15 on Thursday, March 28^th. I’ll be covering a range of important findings, based on my research.

Posted by Glenn Gabe @ 8:43 am in facebook, google, seo, social-media

20 Comments »

Thursday, March 21st, 2013

How To Properly Demote Sitelinks in Google Webmaster Tools

How to remove sitelinks in Google Webmaster Tools

I’ve received several questions recently about how to remove sitelinks in Google. If you’re not familiar with sitelinks, they are additional links that Google provides under certain search listings. Sitelinks enable users to drill deeper into a site directly from the search results. You typically see sitelinks for branded searches.

For example, here are sitelinks for Amazon:
Sitelinks for Amazon.com

And here are sitelinks for the Apple iPad:
Sitelinks for Apple iPad

How Google Determines Sitelinks
Google algorithmically determines sitelinks for a given query/url combination. This is based on a number of factors that Google takes into account. For example, Google explains that it analyzes a site’s link structure to determine if there are additional links it can provide in the search results that will save users time (by quickly enabling them to link to core pages on your site). Remember, Google always wants to connect users with the information they are seeking as fast as possible.

No, Google Doesn’t Always Get It Right
If you are checking your rankings and notice strange sitelinks showing up, you can always demote those links via Google Webmaster Tools. For example, you might see sitelinks that are irrelevant, too granular, or links that could end up sending users to supporting pages that wouldn’t provide a strong user experience. Whatever the case, you can take action.

For cases like this, you can use the “Sitelinks” section of Google Webmaster Tools to demote specific sitelinks. Note, if you don’t have Google Webmaster Tools set up for your site, stop reading this post, and set it up NOW. You can set up your account and verify your site in just a few minutes, and then you’ll receive a boatload of important data right from Google.

Demoting Sitelinks in Google Webmaster Tools
Once you set up a webmaster tools account, you can access the sitemaps section to begin demoting specific sitelinks. Below is a step by step tutorial for demoting sitelinks that shouldn’t be showing up below your search listings.

1. Access the Sitelinks Section of Webmaster Tools
Access Google Webmaster Tools and click the “Configuration” tab, and then “Sitelinks” to access the demotion form.

How to access sitelinks in Google Webmaster Tools

2. Choose Wisely When Demoting Sitelinks
There are two text fields you need to concern yourself with in the “Sitelinks” section. The first is labeled, “For this search result:” and it refers to the webpage that shows up in the search results that contains sitelinks. I know this is where confusion starts to set in, so let me say that again (and show you what I mean).

The first text field is not for the sitelink URL you want to demote. It’s for the webpage that the sitelinks show up for. It’s the URL that’s displayed at the top of the search listing. Note, if you are demoting a sitelink for your homepage, you can leave this field blank. It’s also worth noting that Google provides the root URL of your site already in the text field, so you just need to worry about the remaining part of the URL, which is called the URI (everything after http://www.yourdomain.com/).

For example, if you were the VP of Marketing for Apple, and wanted to remove the “Refurbished iPad” sitelink for the iPad page, then you would enter http://www.apple.com/ipad/ in the first field.

How to remove sitelinks for the ipad search result.

3. Demote the Sitelink URL
The second field is where you will enter the URL of the sitelink you want to demote. Using our Apple example above, you would enter http://store.apple.com/us/browse/home/specialdeals/ipad in the field to demote the “Refurbished” sitelink for the ipad URL. That’s the refurbished iPad page on Apple’s site (and it’s where the sitelink in the search results points to).

Once you enter the URL, you can click the red “Demote” button. Once you do, the demoted sitelink will be listed below the form with the search result it applies to, the specific sitelink URL, and a “Remove Demotion” button. If you ever want to remove the demotion, just access this page again, and click “Remove Demotion”. Then give Google a few days to apply the changes.

Enter the sitelink url to demote.

Misc. Sitelink Information
Based on the questions I have received when helping clients demote sitelinks, I figured I would provide some additional information in this post.

1. How Long Does it Take for Google to Demote Sitelinks?
I’ve seen sitelinks get demoted in just a few days. That said, it definitely varies per site… I’ve seen it take a little longer in certain cases. I recommend monitoring the sitelinks for the page in question for at least a week or two after demoting a sitelink. If you notice that it’s still showing up, then revisit the form to make sure you demoted the right sitelink for the right search result.

2. How Many Sitelinks Can I Demote?
You can demote up to 100 URL’s via Google Webmaster Tools. That should be plenty for most webmasters. Actually, I’d argue that something is very wrong if you are demoting too many sitelinks… You might want to analyze your internal navigation, including the anchor text, to see if Google is picking up something that it shouldn’t be.

Summary – Demotion Can Be A Good Thing
I hope this tutorial helped you better understand what sitelinks are and how to address the wrong sitelinks showing up in the search results. If you notice any weird sitelinks showing up in the search results for your site, then visit Google Webmaster Tools and demote those specific sitelinks. It’s one of the few times that a demotion could be a good thing.

Posted by Glenn Gabe @ 9:16 am in google, seo, tools

No Comments »

Wednesday, February 27th, 2013

Enhanced Campaigns and Local Search | How To Use Bid Adjustments with Radius Targeting to Customize Bids Per Location [Tutorial]

Enhanced Campaigns in Google AdWords

On February 6^th, Google AdWords released a major update to its platform called Enhanced Campaigns. Advertisers can upgrade to enhanced campaigns now (as an option), but all campaigns will be upgraded by mid-2013. Enhanced campaigns provide a radically different way to target by device, adjust bids for various targeting options, set up versatile ad extensions, etc. Therefore, if you’re running paid search campaigns in AdWords, you should get up to speed sooner than later.

The major changes to AdWords with enhanced campaigns include the ability to combine targeting methods in one single campaign versus having to break out separate campaigns. For example, in the past, it was optimal to separate your desktop and mobile campaigns. This would enable you to tailor creative, drive users to mobile optimized pages, bid differently, etc.

In addition, if you were targeting various locations, you would also break those out to different campaigns. For example, you might have had one campaign targeting New York and another New Jersey (in separate campaigns). Using enhanced campaigns, you can use bid adjustments to control bidding per location versus having to break them out to additional campaigns.

Those are just two of the changes that enhanced campaigns bring, and you should definitely read up on the rest. Again, you will be forced to upgrade at some point, and you don’t want to frantically get familiar with enhanced campaigns at the last minute.

Local Businesses and Radius Targeting
So, how do enhanced campaigns impact location targeting for local businesses? Well, if you are targeting users located near your physical location, then enhanced campaigns provide a great way to control bids by location. Specifically, you can use radius targeting to adjust bids based on the distance from your store, office, etc.

Radius Targeting in Google AdWords

For example, let’s say you ran a plumbing service in the Princeton Area. Since you need to travel to customer locations, and many need help immediately, you know that prospective customers located within 15 miles of your office are highly valuable targets. But, you also know that prospective customers as far out as 30 miles can work for your business too. But, time equals money, and 30 miles out isn’t ideal when you factor in travel time. Again, this is all hypothetical, but you get the picture.

So, based on this situation, you could use enhanced campaigns to target users 15 miles out and 30 miles out, and set bids accordingly (using bid adjustments). A bid adjustment lets you increase or decrease your bids, based on certain criteria. In this case, you might set a standard bid that your 15 mile radius targeting could use, and then decrease your bid for users located up to 30 miles away. Then AdWords would dynamically adjust your bids based on the location of the person searching for your services.

Below, I’m going to show you how to set this up in an enhanced campaign. Note, you would obviously need to first upgrade your campaign to “enhanced” before being able to implement the steps listed below. Google has provided an upgrade guide in case you want to upgrade one of your campaigns prior to following the steps below.

How to Use Radius Targeting in Enhanced Campaigns (with Bid Adjustments):

1. Access one of your campaigns that has been upgraded to an “enhanced campaign” and click the “Settings” tab.

Campaign Settings in Enhanced Campaigns

2. Click the “Locations” tab within your campaign settings. Then click “Edit locations”.

Location Tab in Campaign Settings (in Enhanced Campaigns)

3. This is where you can add locations to target. In the field labeled, “In what locations do you want your ads to appear”, enter the city or zip code you want to target. For example, I will enter “Princeton, NJ”. AdWords will autopopulate locations based on what you’re entering. When you see the city you want to target, click the “Nearby” link.

Adding a location to target in enhanced campaigns.

4. Click “Radius Targeting” in the top row of links and re-enter the city and state combination. Then choose a distance away from your location for the first target location. Let’s enter 15 miles and click “Search”.

Using Radius Targeting in Enhanced Campaigns

5. AdWords will show the custom location underneath the radius targeting field. You can then click “Add” to add that location as a target for your campaign. Click “Done” at the bottom of the location targeting window after you have added the first location.

Adding Radius Targeting in Enhanced Campaigns

6. But we’re not done yet. We need to add a second targeting option for 30 miles away. Then we’ll adjust bids based on location. You should see your first location in the “Edit Locations” window (with the radius targeting you just selected). You should only see one location at this point. In order to add another location target, click the “Nearby” link next to the first location target. This will bring up the location targeting window again.

Adding a second location target in enhanced campaigns.

7. Similar to what we did earlier, click the “Radius Targeting” link in the top list of links. Then enter your city and state combination, or zip code, and enter a radius. This time, we will enter 30 miles. Remember, we want to still reach users up to 30 miles away, but we don’t want to pay as much per click. Once you click “Search”, AdWords will show your location with the radius you selected. Then simply click “Add” to add that as a second location target. Click “Done” at the bottom of the location targeting window after you have added the second location.

Adding a second location target using radius targeting in enhanced campaigns.

8. Below the “Edit Locations” window, you should see your two location targets (one for 15 miles out and the other for 30 miles out). You should see a column for “Bid adj.” which will enable you to include bid adjustments per target. Our strategy is to bid normally for users located up to 15 miles out from our location, but we want to lower that bid by 30% for users located up to 30 miles out.

To do this, simply click the dashed line in the Bid adj. field for the 30 mile target and select “Decrease by” and then enter 30 in the percentage field. Click “Save” to lock in the bid adjustment for the location target.

Adding a bid adjustment in an enhanced campaign.

Including bid adjustments for location targets in enhanced campaigns.

That’s it! You just set up a smart scenario for bidding normally for users located closer to your location, while lowering your bids for users outside your typical service zone. Dare I say we’ve just covered an enhanced way to bid. :)

Summary – Use the (Local) Power of Enhanced Campaigns
What I explained above is just one way for local businesses to take advantage of enhanced campaigns. You can also mix various targeting methods with bid adjustments to create advanced-level targeting scenarios. For example, combining desktop, mobile, and location to ensure you are bidding appropriately for the right devices, and for the right locations. But that’s for another day, and another tutorial.

For now, I recommend familiarizing yourself with enhanced campaigns, and testing various location targeting scenarios. Use bid adjustments by location to ensure you are bidding correctly, based on the user’s location. Then you can expand from there. Good luck.

Posted by Glenn Gabe @ 2:09 pm in adwords, google, sem

No Comments »

Tuesday, January 22nd, 2013

Facebook Graph Search – My Initial Thoughts on Speed, Privacy, Reputation Management, Not Provided, and more

Last week, Facebook unveiled Facebook Graph Search, which is an important step forward for the social network. Many in digital marketing, including myself, wondered why Facebook had never added a serious search feature to its platform. It made no sense… Until now, Facebook’s search functionality has been horrible. I often joked that you couldn’t find yourself via Facebook search… :)

By the way, I also wrote a post in October titled “BeastRank”, after Mark Zuckerberg hinted at a Facebook Search Engine. You should check out that post to find out the various ways that Facebook could rank its social search results (based on the enormous amount of data it knows about all of us).

Now that Graph Search has arrived, Facebook finally has the opportunity to better monetize its massive user-base (eventually). Note, Graph search is only in an early form and will absolutely evolve over time. And no, it is not being monetized yet. But you better believe that’s in the works.

What I’m Going to Cover
Facebook Graph Search is definitely an interesting take on social search. I’ve been testing it heavily over the past few days and will cover several of my findings in the post below. You can currently search for people, places, photos, and interests, and I dug into those areas in detail over the past few days.

Note, I’m not going to cover Graph Search basic functionality, or a tutorial on how to use the new search features. There are plenty of posts covering the basics. Instead, I’m going to cover certain aspects of Graph Search that relate to privacy, adoption, SEO, speed, etc. Let’s jump in.

Autocomplete and the Facebook Graph Search Funnel
Once you start using Graph Search, you’ll notice that there are autocomplete suggestions as you start to type. These are essential for Facebook, since it’s not 100% intuitive how you are supposed to structure queries using Graph Search. All of us are used to Google, where you can basically enter anything and have the search engine return results in seconds. I found myself wanting to enter the exact triggers that would surface the right Facebook data. That could be a challenge for Facebook.

If you perform a search that would trigger Graph Search results, then Facebook will offer its guidance. For example “Restaurants in {location}” will auto-populate with locations based on your profile, and will include the Graph Search icon for restaurants next to the suggestion. If you search outside of Graph Search, Bing’s autocomplete takes over (since Bing provides backup for Graph Search).

After performing hundreds of searches over the past few days, I found that Facebook definitely helps you out and starts to rework your searches to fit the Graph Search lingo. For example, if you simply enter a location and restaurants, Graph Search changes that to the “Restaurants in {location}” format. I think the average person might be confused with what they can search for, and how to best surface the information. I know I found it a little confusing, and I’m neck deep in search… That makes me think my mom will have no idea what she is doing. :)

Speed is Important, and Delays Matter
One thing I noticed is that the speed that Facebook returned the results lagged sometimes. And when you are used to a fraction of a second with Google, a few seconds (or longer) felt like a lifetime. I think it’s just another example of how we take Google for granted. If you spend some time in front of Graph Search, I think you’ll appreciate the speed at which Google moves (and for extremely advanced queries).

There were several times that Facebook just hung after I conducted search… I literally viewed a blank screen for a while until the results were returned. Again, this will kill the user experience for Facebook Search, who will undoubtedly leave for Google’s near-instant results.

Local Graph Search – Places, Restaurants, etc.
I found place search pretty cool, but often ran into situations where Facebook didn’t have enough data to provide thorough results. Just a straight search for “Restaurants near Princeton” returned some good results (along with social annotations). I saw a lot of my favorites on the list, but I didn’t need Facebook to know them… There’s a big difference right now between Graph Search and just using Yelp’s app on my phone. Yelp has a ton of data, reviews, etc. Graph Search seems lacking certain areas, and especially outside of the core categories like Restaurants.

If I searched for “restaurants liked by my friends”, I started to see some results that were interesting. I just might want to take a closer look at several of those restaurants. And that is the power of Graph Search, if it can consistently provide recommendations based on my friends’ data. But, there needs to be enough data, and that information needs to be helpful. That’s different than simply returning restaurants that my friends “liked”.

From the results page, I can like the restaurant, view it on a map, view photos taken by people, and also view visitors of the restaurant. This is where some privacy concerns popped up… Good segue.

Places and Privacy (Photos and Visitors)
When you click “photos in” from the local search results page, you will see all photos taken at the restaurant (or place). By the way, almost all of those photos are from people I don’t know, I’m not friends with, etc. Sure, the photos were shared publicly, but I couldn’t help but question if those people even knew the photos were shared publicly. And now the photos are available to anyone searching Facebook (via Graph Search).

For some restaurants, there were hundreds of photos of people and I could easily see their names, click through to their profiles, like their photo (creepy), view comments and likes, etc. Again, I’m not sure all of those people understand this is happening. Oh yeah, there were many photos of kids too… Way to go parents. You just posted public photos of your kids for the world to see. Ugh.

Visitors and Privacy
If I click “Visitors” when viewing a list of restaurants, Facebook will show me people that visited the restaurant (via check-ins). On that page, I can add them as a friend, message them (creepy), or view their photos, friends, and interests. There is also a faceted navigation that lets me drill into visitors by employer, city, school, hometown, etc. Yes, even more concern over privacy here.

Faceted Navigation, Graph Search-Style
When using Graph Search, you can often drill deeper into the search results via a faceted navigation. For example, when you search for a place, the navigation is present by default and enables you to fine-tune your search. When I searched for, “Restaurants near Princeton my friends like”, the faceted navigation enables me to choose other types of places like bars, hotels, gyms, parks, etc.

I can also choose which friends I want to focus on (if I know certain friends like similar things). I can also change the location, select “visited by” and then choose a friend.

Cool, But How Many Will Use The Faceted Navigation?
If you like shopping online and enjoy using a faceted navigation to refine your searches, then you’ll like this functionality. I definitely found it useful and quickly began digging into more Facebook data. That said, I’m not sure how many people will use it like I did… When you think of the average user (who is spoiled by Google doing everything for them), they might not want to refine their searches. They might expect Graph Search to be smart enough to refine it for them. Time will tell how many people use the faceted navigation in Graph Search.

Online Reputation Management (ORM) and Graph Search
I’ve worked on a lot of online reputation management projects over the years, so I started to dig into that aspect of Graph Search. If you type someone’s name, autocomplete first tries to display your friends’ names. But as you select a friends’ name or complete their name in the search bar, the autocomplete suggestions focus entirely on that person. You’ll see “friends of”, “photos of”, etc.

“See more”, and I mean more…
So, Graph Search provides autocomplete suggestions for specific people, but what happens when you click the “see more” link at the bottom of autocomplete? Well, when you click the “see more” link, autocomplete suggestions prompt you to search for all sorts of interests from the person in question. That includes music, sports, pages, books, groups, apps, etc.

That’s cool, but what’s that further down the list? Oh, religious views and political views! “Danger, Will Robinson!” At some point in your life, you learn that politics and religion are extremely polarizing topics. Well, how about if the world could easily drill into that data about you? Using Graph Search, they potentially could…

So, if you don’t have your privacy settings set up correctly, the world can now view your political and religious affiliations. If you’re cool with that, fine. If you just fell out of your seat in horror, then go make changes now. :) And to make drilling into the data even easier, there’s a nice dropdown on the page that enables me to find out the religious and political affiliations of any of my friends.

Graph Search and “Not Provided” – Hiding the Referrer
As I mentioned earlier, Graph Search falls back to Bing’s search results when it can’t meet a certain query. But how does that data get passed to the destination site? Will we begin to see searches from Graph Search in our analytics packages, and can we view the keywords people are searching for? That would be logical, right? I mean what kind of search engine would hide keywords from webmasters?? :)

Well, you aren’t going to be happy with the answer. I checked the referrer leading from Facebook Graph Search to websites, and it unfortunately looks like any other visit from Facebook. So I guess we have our own version of “not provided” from Graph Search. :) Come on Facebook! Pass the query along in the referrer… SEO’s will love you for it. Until they change that, you will see a standard referral from Facebook even when that person used Graph Search.

Click the image below to see a larger version of the referrer:

It’s worth noting that Facebook Graph Search almost always keeps you within Facebook. It’s only when the results fall back to Bing that you get standard search results that take you off the site. That’s when I think Facebook should send the referrer.

Summary – Pay Attention to the Details
Graph Search is big news, even if this beginning state isn’t perfect. Sponsored results (advertising) can turn into a lot of revenue for Facebook, as long as the search experience is strong and the results are compelling. But since it’s social in nature, and based on our data, Facebook has to be really careful with privacy. Based on my research so far, I wouldn’t give Graph Search outstanding scores on that front. In addition, the user experience needs to be faster, and improve, in order to meet the Google standard. Sure, it’s a new twist on search, but Facebook can’t take a step back with performance.

Needless to say, I’m eager to see how Graph Search evolves. And I’ll be closely watching adoption, user experience, privacy, the quality of the search results, etc. I’ll be writing more posts on the subject in the coming months, so definitely check back often. Now excuse me while I find a Chinese restaurant near Princeton that my friends like. :)

Posted by Glenn Gabe @ 5:22 pm in facebook, google, local, social-media

8 Comments »

Friday, January 4th, 2013

Faulty Redirects, Duplicate Content, and SEO – How a Redirect Glitch Created Hundreds of Thousands of Duplicate Pages

Redirect Glitch Causing SEO Problems

With the release of index status in Google Webmaster Tools, many webmasters are now questioning why their “not selected” numbers are high. They wonder if those numbers are good, bad, normal, etc? Unfortunately, there’s not an easy answer to that question, since it depends on the site at hand. But, you can definitely look at the ratio of “not selected” to pages indexed to start to understand if there is a technical problem causing a spike in pages being categorized as “not selected”.

For example, if you have 200 pages indexed on your site, and you see 350 categorized as “not selected”, that might be ok. But if you see 25K pages as “not selected” or more, then that could raise a red flag that something may not be right with the site… For example, is there a site structure issue that’s causing thousands of variations of pages with extremely similar content (duplicate content)?

A Recent Example of a Poor “Not Selected” Ratio
During SEO audits, there are times I come across significant problems like the one I mentioned above. And those problems could be inhibiting a company’s search efforts (to say the least). During a recent SEO audit, I came across a very interesting situation. Index status revealed an extremely high number of “not selected” pages (as compared to the number of pages indexed) and I found myself digging into the site to find out why.

High Percentage of "Not Selected" Pages in Index Status

I found several issues causing the problem, so there wasn’t just one issue pumping up the number. That said, the problem I’m going to cover today was causing thousands of duplicate pages to be created, and without the site owner knowing. The more pages I checked, the more duplicates I found. And this is a problem that can easily slip through the cracks for many webmasters. And that’s especially the case if a small or medium sized business is handling all website development on its own. Below, I’m going to cover what I found, and more importantly, how you can avoid the problem in the first place.

The Danger of an Extra Character
As I was analyzing the site manually, and via a number of test crawls, I came across some URL’s that contained an extra character. Specifically, the extra character was being appended to each canonical URL. All of those URL’s were from one specific section of the site (which contained thousands of URL’s). After digging into that section of the site, I found out that this problem was happening to almost every URL being linked to from a certain element within each page. So, I honed in on that element within each page to find out how the duplicate pages were being created. And by the way, it just so happens that the section of the site contains nearly 200K pages. Yes, this was a huge problem that was uncovered.

How One Extra Character Created Thousands of Duplicate Pages

The Result – Duplicate Content Ad Infinitum
The core problem is that the extra character created a new URL, but that new URL was an exact duplicate of the canonical URL (the URL that should be resolving). And as you can guess, both pages can be accessed on the site. One part of the site links to the canonical version of the pages, while this problematic section linked to the duplicate versions of the pages.

So, right off the bat, we are dealing with at least 200K duplicate pages. In addition, as more content is added to this section, more duplicate pages will be created over time (based on the extra character being added to each URL). Also, the canonical URL tag was not being used on the duplicate pages, so that wasn’t helping this specific case. And on that note, I wouldn’t advocate using the canonical URL tag to fix this problem… Technical problems like this should be addressed at the code or structure level.

1 to 1 Ratio of Duplicate URL's to Canonical URL's

So, if this was left in place, this problem could generate an unlimited number of duplicate pages. If 500K pages ended up there, then there would be 500K pages of duplicate content. Not good, so I dug deeper to find out exactly what was causing the problem.

The Root Problem – Faulty Redirects
Let’s face it, we all need to implement redirects at some point. And that introduces the possibility of a poor implementation, which can be catastrophic SEO-wise. It’s one of the reasons that website redesigns and CMS migrations are so risky. On that note, to learn how to avoid SEO disaster during a redesign or migration, you should check out my Search Engine Journal column on the subject.

For example, using 302’s versus 301’s, using meta refresh redirects, redirecting to the wrong pages, or having the redirect code bomb the URL’s. For this situation, we ran into the “bombing of URL’s” problem. The redirects were faulty, and were redirecting to URL’s with an extra character.

The Solution – Fix the Redirect Code!
So, hundreds of thousands of duplicate pages were being generated, and it was due to one piece of redirect code on the server. The 301 redirects being generated simply added an extra character to the destination URL. That’s it. The fix will be implemented soon, and once the new redirect code is rolled out, the correct URL’s will resolve.

This situation underscores the fact that even one small piece of code could have serious implications SEO-wise. If this situation was left unchanged, it could have ended up generating an unlimited number of duplicate pages. Knowing the content on this site, my guess is the problem would have generated 500K-750K pages of duplicate content over the next 2-3 years.

How To Avoid This Situation
After reading this post, you might be scared that this could happen to you, or worse, that it’s happening right now. I’m going to provide a short list of things you can do to make sure this doesn’t happen. Of course, if you feel you are having problems already, you should have an SEO audit performed.

First, whenever you create redirects, make sure you have a system for testing those redirects before they launch. You can do this a number of ways, including on a local server or test server prior to releasing the final code to production. If you thoroughly test the redirects, you could nip serious problems in the bud.
Second, make sure your xml sitemaps contain the canonical url’s for the pages at hand. Making sure you are feeding Google and Bing the correct URL’s can help them understand which ones should be considered the canonical url’s.
Third, you should develop a strategy for using the canonical URL tag on the site. If the tag is present, then you can ensure that any duplicate pages pass their search equity to the canonical URL’s. Note, I’m not saying that you should leave a technical problem in place! Instead, I’m saying that having the canonical URL tag in place will make sure the engines pass any search equity to the correct pages on your site while you figure out solutions to your technical problems.
This final bullet assumes you are already experiencing problems with duplicate content from a technical problem. If you are, and you cannot determine what’s going on, then invest in having a technical SEO audit completed. To me, the provide the most bang for your SEO buck. It’s a great way to find out what’s truly going on with your site (beyond just the problem I listed here).

Summary – Know Your Site
This case emphasizes something I’ve said a thousand times over the past few years. It’s incredibly important to have a sound site structure in order to perform at your highest level SEO-wise. Coding problems, site structure issues, flawed redirects, etc. can kill your SEO efforts. It’s one of the reasons that I believe SEO audits are critically important. They can catch all types of SEO issues and provide ways to remedy those problems. You know, like generating hundreds of thousands of duplicate pages. :)

Posted by Glenn Gabe @ 10:38 am in google, seo

2 Comments »

Thursday, December 27th, 2012

Introducing SEO Bootcamp Princeton, A Hands-On SEO Training Course in Princeton NJ

SEO Training Topics

I absolutely love getting in front of a group of people to speak about SEO (and always have). Over the past several years, I’ve led SEO training classes for clients covering a wide range of topics, from technical SEO to keyword research to content optimization to linkbuilding strategy. Although I’ve really enjoyed leading classes like this, I’ve always wanted to launch a training program that anyone could sign up for, and not just clients. Well, I finally put the program together, and it’s called SEO Bootcamp Princeton.

SEO Bootcamp Princeton is a three hour, in-person training course being held at the Johnson Education Center (at D&R Greenway) on January 17^th, from 9AM to 12PM. You can register online via EventBrite, and there’s a 20% off, early registration discount running through 12/31/12. If you register by then, tickets are $145 versus the standard price of $179.

The Target Audience for SEO Bootcamp Princeton
So, what will you learn at SEO Bootcamp Princeton? Put simply, you’ll learn a lot. My goal is to make sure attendees can leave the training ready to make changes to their websites. I’ve crafted the training so it can be valuable for any person marketing a business, from small business owners to corporate marketers. SMB’s will learn the tactical knowledge necessary to build a solid SEO foundation, while corporate marketers can learn SEO best practices and techniques.

In addition, the training will be extremely valuable for creative professionals, including designers, programmers, copywriters, etc. I used to work for a large agency in New York City, and I led a similar type of training there. I can tell you that every creative professional left the training with a stronger understanding of Search Engine Optimization (SEO). Actually, I know the training changed how some people performed their jobs on a regular basis…

For example, designers and programmers learned about search engine friendly ways to design and code sites, while copywriters learned how to perform keyword research and properly optimize content. Professionals involved with information architecture (IA) learned how to best structure a navigation, while also learning the best ways to build an internal linking structure. And everyone in the training learned about the risks of redesigning a website without taking SEO into account.

Those are just a few of the SEO topics you’ll learn more about at SEO Bootcamp Princeton. Again, my goal is that you leave with a much deeper knowledge of SEO, that you can make changes immediately, and that you take SEO into account whenever working on a website, campaign, or redesign. You can learn more about the topics I’m going to cover on the SEO Bootcamp Princeton webpage.

SEO Bootcamp Princeton is Job-Agnostic – All Levels and Positions Will Benefit

Technical and Creative Job Titles

Tools and Plugins
SEO is definitely a mix of art and science. And in order to assist SEO professionals with several core tasks, there are many tools and plugins one can use. During the training, I will highlight several of the tools and plugins that can make your job easier SEO-wise. I’ve always said that when you combine the right tools with the right SEO knowledge, great things can happen. And I’ll make sure to explain some of my favorites along the way. From Firefox plugins to Chrome extensions to standalone software applications, you’ll leave the training with a list of tools that can help you on a regular basis.

SEO Tools Training

Major Algorithm Updates
I can’t leave this post without touching on a very important topic in SEO that’s affecting many business owners. Google has launched several important algorithm updates since early 2011, including both Panda and Penguin. As you can imagine, I receive calls every month from business owners that have gotten hammered by these updates. During SEO Bootcamp Princeton, I will introduce each major algorithm update, and cover important insights based on helping a range of businesses deal with the aftermath of getting hit. And more importantly, I can explain the best ways to avoid getting hit in the first place. You can read several of my case studies about Panda recovery and Penguin recovery if you are interested in learning more.

Panda and Penguin Algorithm Updates

Next Steps, Register Today
In closing, I’m ultra-excited about SEO Bootcamp Princeton. If you are interested in registering, you can sign up via the EventBrite page. Again, there’s a 20% off, early registration discount running through 12/31. After 12/31, the standard pricing will be $179 per seat. If you have any questions about the training, don’t hesitate to contact me. It would be great to see you there!

Posted by Glenn Gabe @ 10:18 am in google, seo, tools

No Comments »

Tuesday, November 27th, 2012

How Google Analytics Really Handles Referring Traffic Sources [Experiment] – Why Clicks and Visits Might Not Match Up

Google Analytics Referrals

Let me walk you through a common scenario in web marketing. You have a website, and some people visit your site by clicking through links on other websites. In your web analytics reporting, those visits are categorized as referring visits. In Google Analytics specifically, those visits show up in your “Referrals” report under “Traffic Sources”. And when visitors click on an outbound link on your site (a link to another website), your site shows up as a referring source in that website’s referrals report.

That’s pretty straight forward, but what I’m about to cover isn’t. I find many marketers and webmasters don’t understand how Google Analytics handles that referring traffic during future visits to their websites. For example, if someone clicks through to your site from sampledomain.com, leaves your site, and then returns the next day.

Do you know how that visit will be categorized in Google Analytics? There’s a good chance you don’t, and I’m going to cover the topic in detail in this post.

Understanding Referring Visits is Important When Revenue and Cost Are Involved
I believe one of the reasons this topic isn’t understood very well is because it often doesn’t directly impact revenue or cost for many webmasters. Sure, you definitely want to know how many people are coming from each referring site, but for many webmasters, the exact number doesn’t impact revenue, or payments to other webmasters.

But, for websites that need to track the monetary value of inbound visits and outbound clicks, accurately determining referring visits is extremely important. For example, imagine you were charging certain partners for traffic you were sending from your site to theirs, or vice versa. The fact of the matter is that checking referring sources could show different numbers than you think, and could be much different than the outbound clicks you see. And depending on your own situation, the numbers could be way off…

The Core Disconnect – How Google Analytics Calculates Visits from Referring Sources (or any campaign, search visit, etc.)
Here’s the core disconnect. When someone clicks through to your site via a referring source, the utm_z cookie is updated with traffic source information. That cookie will not be overwritten unless another referring source or campaign takes it place. Direct Traffic will not overwrite this value. Let me say that again. Direct Traffic will not overwrite the utm_z cookie value. That means the utm_z value will remain the referring source of traffic when those visitors return to the site.

Google Analytics utm_z Cookie

What This Means To You
I know what you’re thinking. This guy is telling me about utm_z cookies?? What the heck does that mean to me? OK, stick with me for a second. Let’s say you had a partnership set up where another website pays you for traffic. Maybe you’re both in the same niche and want to leverage each other’s traffic for more exposure. You check your stats for the previous month and notice that you sent 500 visits to partner A. Cool, so you contact them to check how the partnership is going and to make sure they are seeing the same number of visits. They come back and say they’ve seen 700 visits from your site and thank you for the traffic. The check will be cut soon.

Google Analytics Clicks and Visits Could Be Off

But that 200 visit discrepancy is bothering you. Why is there a big difference between your partner’s reporting and the numbers you are seeing? And let’s assume you have a solid setup for tracking clicks out of your website. For example, maybe you are running outbound clicks to partners through a redirect that captures a number of important metrics. The redirect then sends the visitor off to the correct URL on the partner site. Basically, you know you are capturing all outbound clicks to the partner website.

This is where the native handling of referring sources in Google Analytics comes into play. Sure, you are tracking clicks off your site, but your partner’s analytics package is capturing those clicks plus any return visits that are direct visits. So, if someone clicks through to your partner’s site, then that’s one visit. If they leave that site, and return directly (by typing the url directly in their browser or via a bookmark), then the visit will show up as a visit from the original referring source (your website). That’s now two visits. And if they do it again, that will be three visits. That’s until another referring source or campaign overwrites the utm_z cookie. In this example, there were 3 visits to your 1 outbound click!

An Example of How Google Analytics Handles Referrals

Based on this simple example, you can easily see how over a month’s time, some people would click through to your partner’s site and then revisit their site directly (and possibly a few times). That would lead to more than one visit per user, and could sway the visit count from your website.

Still confused? Let me clear this up via an experiment below.

Experiment – Calculating Referring Visits in Google Analytics
In the following simple example, I set up a webpage on a second domain that links to a landing page I set up on my website just for this experiment. I didn’t want to skew the reporting by using an existing page on my site that gets a lot of visits. Then I used several computers I have here with clean browsers to first visit the referring page that links to my new landing page, and then I clicked through. The referring source should show up as the domain name of the referring site. That would be visit #1.

Next, I would leave the new landing page on my site and revisit my website later by typing the exact URL into my browser (what most people would think is a Direct Traffic visit). In theory, the referring site should show up as the traffic source, even though I’m entering the site as “Direct Traffic”. Remember, the utm_z cookie will only be overwritten by another referring source or campaign.

Last, I would search for a keyword that my site ranks for, and then click through to the site. And since this visit was from a search engine, the utm_z cookie would be updated with this new value, and my reporting would show Google as the referring source (along with the keyword I entered). Let’s find out the results of the experiment below.

The Results
1. First Visit
First, I visited the second domain and clicked through to my website. Here is the first referring visit showing up in my analytics package:
Referring Sites Experiment - First Visit

2. Second Visit (Directly Visiting the Site)
Next, I left the site and returned via Direct Traffic. Google Analytics shows the referring site as the source for this traffic, even though I entered via “Direct Traffic”. Also notice it accurately categorizes me as a “return visitor”:
Google Analytics Referral Experiment - Second Visit

3. Third Visit (Again Directly Visiting the Site, but the Next Day)
Just to underscore my point, I left and revisited the site the next day (again via Direct Traffic). Google Analytics again shows the visit is from the initial referring source:
Google Analytics Referral Experiment - Third Visit

4. Fourth Visit, This Time From Search
Finally, I searched for my name on Google and visited my website. Now Google Analytics shows the keyword that led to the site (from the traffic source “Google”). Remember, the utm_z cookie will only be updated when another referring source is identified (versus Direct Traffic).

So there you have it. Proof that your visit count by source may not be what you think it is. Now, if you’re reading this post and are either generating revenue from referring visits, or you have to pay partners based on visits, then you might be frantically running to Google Analytics to rerun your reports. Yes, this could impact things quite a bit. I’ll leave it up to you how you handle the situation. :)

What You Can Do – The Importance of Clarity
If you do have a partnership where you are either generating revenue by driving traffic, or you are paying for traffic from other websites, then each party needs to clearly understand the arrangement. Each website involved needs to be clear on the definitions of “traffic”, “clicks”, “visits”, etc. For example, think about AdWords for a second. You pay Google for clicks on ads, but don’t pay Google for direct visits back to your site (even though those visits will show up as campaign visits). And by the way, most partners will not give you access to their reporting anyway… Therefore, you will only know the clicks out from your site.

If you are tracking outbound clicks, you can use event tracking in Google Analytics to track those clicks, including the pages or links where those clicks are originating. If you don’t want to use event tracking, then you can run outbound clicks through a 302 redirect and capture the information you need to accurately track clicks. If you are receiving traffic, then you can make sure the referring links contain querystring parameters so you can understand which partner the traffic is coming from (and that it’s not a standard referral from the site). There are other ways to handle this, and those are just a few ideas.

Summary – Understanding Visits in Google Analytics
I hope you found this post explaining how Google Analytics handles referring visits helpful. I know this topic can be confusing, and experiments always help clear up some of the confusion. So now you know why visits might be higher or lower than you think, and how the utm_z cookie controls what shows up in your reporting. I bet you’ll never look at referring sources the same again.

And let’s hope you’re not on the short end of the stick. :)

Happy Reporting.

Posted by Glenn Gabe @ 11:14 am in google, google-analytics, web-analytics

9 Comments »

Wednesday, November 14th, 2012

Hunting False Negatives – How To Avoid False Negatives When Checking Redirects After a Website Redesign or Migration [Screaming Frog Tutorial]

How To Check Redirects Using Screaming Frog

Every webmaster has to deal with a website redesign or migration at some point. And redesigns and migrations often mean that your URL structure will be impacted. From an SEO perspective, when URL’s need to change, it’s critically important that you have a solid 301 redirection plan in place. If you don’t, you can pay dearly SEO-wise.

I wrote a post for my Search Engine Journal column last spring titled “How to Avoid SEO Disaster During a Website Redesign” and implementing a 301 redirection plan was one of the most important topics I covered. I find many webmasters and marketers don’t understand how SEO power is built URL by URL. As your URL’s build up inbound links and search equity, it’s important that those URL’s maintain those links and equity. If you change those URL’s, you must notify the search engines where the old content moved to, and that’s where 301 redirects come into play.

So, when you change URL’s, you run the risk of losing all of the links pointing to the older URL’s, and the search power that the URL’s contained. That’s unless you 301 redirect the old URL’s to the new ones. A 301 redirect safely passes PageRank from an old URL to a new one (essentially maintaining its search equity).

Unfortunately, I’ve seen many companies either not set up a redirection plan at all, or botch the plan. That’s when they end up with a catastrophic SEO problem. Rankings drop quickly, traffic drops off a cliff, sales drop, and nobody is happy at the company (especially the CMO, CFO, and CEO).

Traffic Drop After Website Redesign

Meet the False Negative Redirect Problem, A Silent Killer During Redesigns or Migrations:
Needless to say, properly setting up your redirects is one of the most important things you can do when redesigning or migrating your website. That said, even if you address redirects and launch the new site, how do you know that the redirects are in fact working? Sure, you could manually check some of those URL’s, but that’s not scalable. In addition, just because an older URL 301 redirects to a new URL doesn’t mean it redirects to the correct URL. If you don’t follow through and check the destination URL (where the redirect is pointing), then you really don’t know if everything is set up properly.

This is what I like to call the False Negative Redirect Problem. For SEO’s, a false negative occurs when your test incorrectly shows that the redirects are working properly (they don’t test positive for errors), when in fact, the destination URL’s might not be resolving properly. Basically, your test shows that the redirects are ok, when they really aren’t. Incorrectly thinking that 301 redirects are working properly by only checking the header response code for the old URL can trick webmasters into believing the redesign or migration has gone well SEO-wise, when in reality, the destination URL’s could be 404’ing or throwing application errors. It’s a silent killer of SEO.

False Negatives can be a Silent SEO Killer

How To Avoid the Silent SEO Killer When Changing Implementing Redirects
The false negative problem I mentioned above is especially dangerous when changing domain names (where you will often implement one directive in .htaccess or ISAPI_Rewrite that takes any request for a URL at one domain and redirects it to the same URL at another domain). Just because it 301’s doesn’t mean the correct URL resolves. Think about it, that one directive will 301 every request… but you need to check the destination URL to truly know if the redirects are working the way you need them to. Unfortunately, many SEO’s only check that the old URL’s 301, but they don’t check the destination URL. Again, that could be a silent killer of SEO.

Screaming Frog Hops to the Rescue
I mentioned “scalable” solutions earlier. Well, Screaming Frog provides a scalable solution for checking redirects during a migration or website redesign. Note, Screaming Frog is a paid solution, but well worth the $157 annual fee. Using Screaming Frog, you can import a list of old URL’s from your analytics package or CMS and have it crawl those URL’s and provide reporting. Running a two-step process for checking redirects and destination URL’s can help you understand if your redirects are truly working. For example, you might find redirects that lead to 404’s, application errors, etc. Once you find those errors, you can quickly change them to retain search equity.

Below, I’m going to walk you through the process of exporting your top landing pages from Google Analytics and checking them via Screaming Frog to ensure both the redirects are working and that the destination URL’s are resolving correctly. Let’s get started.

What You’ll Need and What We’ll Be Doing

First, we are going to export our top landing pages from Google Analytics.
Second, we’ll use the CONCATENATE function in Excel to build complete URL’s.
Next, we’ll add the URL’s to a text file that we can import into Screaming Frog.
Then we’ll fire up Screaming Frog and import the text file for crawling.
Screaming Frog will crawl and test those URL’s and provide reporting on what it finds.
Then we can export the destination URL’s we find so we can make sure they resolve correctly. Remember, just because the old URL’s 301 redirect doesn’t mean the destination URL’s resolve properly. We are hunting for false negatives.
Last, and most importantly, you can fix any problematic redirects to ensure you maintain search equity.

How To Use Screaming Frog to Hunt Down False Negatives:

Export Top Landing Pages from Google Analytics
Access your Google Analytics reporting and click the “Content” tab, “Site Content”, and then “Landing Pages”. Click the dropdown for “Show rows” at the bottom of the report and select the number of rows you want to view.

Tip: If you have greater than 500 pages, then you can edit the URL in Google Analytics to display greater than 500 URL’s. After first selecting a row count from the dropdown, find the parameter named table.rowCount= in the URL. Simply change the number after the equals sign to 1000, 5000, 10000, or whatever number you need to capture all of the rows. When you export your report, all of the rows will be included.
Export the Report from Google Analytics
Click the Export button at the top of the report and choose “CSV”. The file should be exported and then open in Excel once it downloads.
Use Excel’s CONCATENATE Function to Build a Complete URL
When the URL’s are exported from Google Analytics, they will not include the protocol or domain name. That’s the beginning of a URL with http://www.yourdomain.com. Therefore, you need to add this to your URL’s before you use them in Screaming Frog. Excel has a powerful function called CONCATENATE, which lets you combine text and cell contents to form a new text string. We’ll use this function to combine the protocol and domain name with the URL that Google Analytics exported.

Create a new column next to the “Landing Page” column in Excel. Click the cell next to the first landing page URL and start entering the following: =CONCATENATE(“http://www.yourdomain.com”, A8). Note, change “yourdomain.com” to your actual domain name. Also, A8 is the cell that contains the first URL that was exported from Google Analytics (in my spreadsheet). If your spreadsheet is different, make sure to change A8 to whichever cell contains the first URL in your sheet. The resulting text should be the complete URL (combining protocol, domain name, and URL exported from Google Analytics). Then you can simply copy and paste the contents of that cell (which contains the formula) to the rest of the cells in that column. The formula will automatically adjust to use the right landing page URL for that row. Now you have a list of all complete URL’s that you can import into Screaming Frog.
Copy all URL’s to a Text File
Since all we want are the URL’s for Screaming Frog, you can select the entire new column you just created (with the complete URL’s) and copy those URL’s. Then open a text file and paste the URL’s in the file. You can use Notepad, Textpad, or whatever text editor you work with. Save the file.
Fire Up Screaming Frog
After launching Screaming Frog, let’s change the mode to “list” so we can upload a list of URL’s. Under the “Mode” menu at the top of the application, click “List”, which enables you to use a text file of URL’s to crawl. Then click “Select File” and choose the text file we just created. Then you can click “Start” and Screaming Frog will begin to crawl those URL’s.
Review Header Response Codes From the Crawl
At this point, you will see a list of the URL’s crawled, the status codes, and the status messages. Remember, all of the URL’s should be 301 redirecting to new URL’s. So, you should see a lot of 301’s and “moved permanently” messages. If you see 404’s at this point, those URL’s didn’t redirect properly. Yes, you just found some bad URL’s, and you should address those 404’s quickly. But that’s not a false negative. It’s good to catch low-hanging fruit, but we’re after more sinister problems.
Find the Destination URL’s for Your Redirects
Now, just because you see 301 redirects showing up in the main reporting doesn’t mean the destination URL’s resolve correctly. If you click the “Response Codes” tab, you’ll see the redirect URI (where the 301 actually sends the crawler). THOSE ARE THE URL’S YOU NEED TO CHECK. Click the “Export” button at the top of the screen to export the “Response Code” report. This will include all of the destination URL’s.
Copy All Destination URL’s to a Text File
In Excel, copy the destination URL’s and add them to a text file (similar to what we did earlier). Make sure you save the new file. We are now going to crawl the destination URL’s just like we crawled the original ones. But, this process will close the loop for us, and ensure the destination URL’s resolve correctly. This is where we could find false negatives.
Import Your New Text File and Crawl the Destination URL’s
Go back through the process of selecting “List Mode” in Screaming Frog and then import the new text file we just created (the file that contains the destination URL’s). Click “Start” to crawl the URL’s, and then check the reporting.
Analyze the Report and Find False Negatives
You should see a lot of 200 codes (which is good), but you might find some 404’s, application errors, etc. Those are your false negatives. At this point, you can address the errors and ensure your old URL’s in fact redirect to the proper destination URL’s. Disaster avoided. :)

Screaming Frog and Actionable Data: Beat False Negatives
Going through the process I listed above will ensure you accurately check redirects and destination URL’s during a website redesign or migration. The resulting reports can identify bad redirects, 404’s, application errors, etc. And those errors could destroy your search power if the problems are widespread. I highly recommend performing this analysis several times during the redesign or migration to make sure every problem is caught.

Make sure you don’t lose any URL’s, which can result in lost search equity. And lost search equity translates to lower rankings, less targeted traffic, and lower sales. Don’t let that happen. Perform the analysis, quickly fix problems you encounter, and retain your search power. Redesigns or migrations don’t have to result in disaster. You just need to look out for the silent SEO killer. :)

Posted by Glenn Gabe @ 9:07 am in google, seo, tools, web-analytics

1 Comment »

Wednesday, October 31st, 2012

Dynamic Search Ads (DSA) in Google AdWords – How Your Technical SEO Problems Just Impacted Your SEM Campaigns

In October of 2011, Google began testing Dynamic Search Ads (DSA) in AdWords. It was a bold move and signaled a change in how paid search could operate in the future. Using dynamic search ads, advertisers could greatly expand their reach by enabling AdWords to match queries with content in Google’s organic index. Instead of simply setting up keyword-based campaigns, you could dynamically provide relevant ads to users searching for content residing on your site.

Last week AdWords released Dynamic Search Ads to all users, so now everyone can expand their reach using DSA’s. But before you run and set up your campaigns, there’s a catch you need to be aware of. Your content must be indexed in Google’s organic search index in order for it to be eligible for dynamic search ads. Yes, your SEO just impacted your SEM, and that’s what my post is about today. Read on.

Targeting Dynamic Search Ads (DSA)
Targeting-wise, you can set up dynamic ad targets based on the content that’s been indexed on your site. For example, you can target all webpages on your site, categories of content, pages by URL, content by page title, or simply by content residing on a page. Depending on the dynamic ad targets you set up, AdWords will match up queries with your content that’s been indexed. I bolded those last few words, since thorough indexation can be a big problem for some companies. More on that soon.

You can target DSA’s by Category, URL, Page Title, or Page Content:

Dynamic Ad Targets in Google AdWords

Dynamic Search Ads Example
Imagine you sold shoes and had AdWords campaigns already set up for sneakers and boots. I’m sure you would have more, but let’s keep this simple. Now, let’s say you have 150 specific products that fall into these categories, but aren’t set up in your AdWords campaigns (you just have categories set up). These products are essentially left out in the paid search cold… until now. Using dynamic search ads, you could target those specific products that don’t have campaigns set up and let AdWords match up your content with targeted queries automatically (based on what’s indexed in Google’s organic index). AdWords could dynamically build the ad title, include ad text written by you, and then determine the destination URL based on what’s indexed in Google. Hence the “dynamic” part of Dynamic Search Ads. :)

An example of creating a dynamic search ad in AdWords:

Yes, it’s paid search less the keywords. As you can imagine, this can have a major impact on how paid search is managed, optimized, and enhanced by AdWords advertisers. But more importantly, now technical SEO issues can negatively impact your DSA’s, since they leverage Google’s organic search index. That’s a good segue to the next section of my post.

DSA’s are based on Google’s Organic Search Index (Meet Your New SEO Problem)
As I mentioned earlier, in order for dynamic search ads to work, your content needs to be indexed. Although that sounds trivial, it’s not for some companies. And that’s especially true for large-scale sites with hundreds of thousands of pages (or more). And if you add CMS problems to the mix, a site could very well have tens of thousands of pages not getting indexed properly. And that could be a serious problem for advertisers trying to leverage DSA’s to expand their reach.

If your site has a crawlability problem, or other technical SEO problems, then those problems can now affect your SEM campaigns. In addition, if your competitors don’t have those SEO problems, then they are at a big advantage with regard to dynamic search ads. If they set up their DSA campaigns intelligently, then they can potentially reach a much wider audience than you can, since they have a much deeper set of content indexed.

Indexation Issues Impacting DSA's

Uh Oh, The Holidays Are Here
Did you just start to sweat? Right, the holidays are just around the corner… That’s perfect timing for some companies to boost sales, while others with technical SEO problems falter. In the past, SEM was separate from SEO. But now, SEO has crept into SEM. Let’s take a look at some problems that could cause issues with your DSA campaigns.

7 SEO Problems That Could Affect Your DSA’s

1. Make it Clean and Crawlable, or No DSA’s For You
First, your content needs to be crawled and indexed. I never thought I would be saying that in an SEM-focused post, but there’s a first for everything. :) If your content cannot be easily crawled and indexed, you will be at a major disadvantage with dynamic search ads. Again, AdWords will leverage Google’s organic search index to match content and queries, and then use that content to build dynamic ads. If your content isn’t in the organic index, ads cannot be generated. No DSA’s for you.

Ensure Googlebot can easily crawl your website:

Googlebot Encountering Errors While Crawling a Website

2. Poor Navigation and Internal Linking Structure
An important aspect to getting all of your content crawled is having a strong internal linking structure. There are still many sites that don’t provide a robust drilldown into their content using text links. If you simply provide a top-level navigation and don’t provide additional links to deeper content, then you could easily run into a situation where that additional content isn’t crawled or indexed.

If that’s the case, then that additional content cannot be used for dynamic search ads. I highly recommend reviewing your navigation and internal linking structure to ensure you are providing a descriptive drilldown into your categories, products, articles, blog posts, etc.

Provide a robust drilldown into your content versus hiding it:

Navigation Causing Indexation Problems

3. Gremlins in Your Content Management System (CMS)
In my experience, I’ve witnessed CMS packages hide content, provide serious crawlability issues, and create serious duplicate content problems. And all of this won’t be good for your DSA efforts. If your CMS hides content, then those pages will not be available for DSA’s. If the CMS provides crawlability issues, then it can hide content from Googlebot, which means the content can’t be included in DSA campaigns. And if your CMS generates massive duplicate content problems, then who knows what AdWords will match up with targeted queries (if it will match up any content at all).

Let’s face it, a great CMS can make your life a lot easier. But a poor CMS can wreak havoc on both your SEO and SEM efforts. And now with dynamic search ads, you can combine the two… I highly recommend having your CMS audited to ensure it’s not hampering your search campaigns. I provide more recommendations later in the post.

Understand if your CMS is causing crawlability or indexation issues:

CMS Packages Causing SEO Problems

4. XML Sitemaps
During SEO audits, I still find sites that aren’t providing xml sitemaps that contain all of their content. For example, I recently audited a site with 300K+ pages, but its xml sitemaps only contained 10K. XML sitemaps are a great supplement to a traditional web crawl. You don’t want to rely on them for getting all of your content indexed, but they can help Google identify new content and understand the canonical URL’s for your content. In addition, you can view sitemap errors in Google Webmaster Tools, which can help you understand problems Google is having trying to access or index your content.

For dynamic search ads, XML sitemaps can help more of your content get crawled and indexed by Google. And that can help you target more content via DSA’s. Again, you shouldn’t rely on XML sitemaps to fix crawlability and indexation problems, but it’s a smart move to set them up.

Monitor xml sitemap errors in Google Webmaster Tools:

XML Sitemaps Should Contain All Canonical URL's

5. Poor URL Structure
As mentioned above, you can target content by all web pages, category, URL, page title, or page content. If you want to logically launch DSA’s by URL, then your URL structure needs to be strong. For example, you can target any page with /category/sneakers to target sneakers or /category/boots to target boots. If you have something like /?nid=2343jieejd&sot=23jjdjdj and you are going to try and find some common thread, good luck. Chances are you won’t be able to target by URL. Try and use clean and descriptive URL’s if possible. Stay away from complex URL’s with a lot of querystring parameters.

Complex URL’s can cause crawlability, canonical, and indexation issues:

Complex URL's and Poor URL Structure Can Cause SEO Problems

6. Canonical URL Tag Issues
When used properly, the canonical URL tag can be a great way to address duplicate content issues. You can tell the search engines which pages are the canonical url’s for the content at hand. But when used improperly, it can be extremely destructive to your SEO efforts. I wrote about this in my Search Engine Journal column titled, “Two Examples of How One Line of Code Could Kill Your SEO.” You should read the post when you get a chance. It’s fitting that I’m referencing that post on Halloween, since it’s horrifying. :)

So, if you implemented the canonical URL tag incorrectly across your site, Google may only have a small percentage of your content indexed and available for DSA’s. In a worst case scenario, you could be using the canonical URL tag to attribute all search power to just one page on your site. Don’t laugh, I’ve seen this happen several times. If that’s the case, then you might only have one page available for DSA’s. And you might be looking at your AdWords reporting wondering why there are no impressions or clicks.

7. Poor On-Page Optimization
If you choose to target by page title, then you need to ensure pages are well-optimized. I’ve completed SEO Audits on some larger sites that have thousands of pages with the same exact title tag. If that’s the case, then AdWords might not be able to figure out what the page is about, and might not be able to match the content up with targeted queries. If this is the case, then make sure you uniquely optimize each page, based on the content at hand. If you do, then you can target DSA’s by page title and be in good shape.

Ensure all of your content is uniquely optimized:

Poor Content Optimization Can Impact DSA's

What Can You Do? 3 Things You Can Do Now to Help Your DSA’s
If you are reading this post and determine that you might have some of the SEO problems I listed above, then here’s what you can do now.

1. SEO Audits
First, and this is something I have advocated for a long time, have a technical SEO audit conducted. Audits provide the most bang for your SEO buck. They can be completed relatively quickly and provide a remediation plan based on the findings. If you can implement the changes relatively quickly, then you very well could see some improvements in a short period of time. That obviously depends on your specific situation, but some changes will yield strong results in a short amount of time.

And with the holiday season upon us, time is of the essence. If you want dynamic search ads to help you this holiday season, then you need to make sure your content is being indexed, and that it’s optimized correctly.

2. Index Status in Google Webmaster Tools
Second, analyze Index Status in Google Webmaster Tools, which can help you identify the number of pages Google has indexed, as well as how many it counts as “not selected”. Index Status won’t give you the answers, but can let you know how well your site is being indexed. For example, if you have 10K pages on your site, but only 2K are indexed, you’ve got a problem. If you have 5K pages on your site, but Index Status shows 75K as “not selected”, then you also have a problem. I highly recommend reading my post about Index Status and analyzing your current situation.

Index Status in Google Webmaster Tools

3. Bypass DSA’s and Build Out More Campaigns Manually
Third, if you have indexation issues, but still want to gain additional targeted traffic via Paid Search, then you can focus your attention on fleshing out more campaigns and ad groups based on your own category and niche. Perform thorough keyword research, understand which keywords you need to target, analyze the competitive landscape, and then build out more campaigns and ad groups. Sure, this will take a lot of time and effort, but it provides the most control.

Summary – The End of Keywords in SEM?
Dynamic Search Ads (DSA) can help advertisers reach a broader audience by automatically matching up advertiser content and targeted queries. Paid search without keywords could very well be the future of SEM, so it’s important to understand how DSA’s work now. But as I explained throughout this post, your content needs to be crawled and indexed in order to be eligible for dynamic search ads. And that means SEM will require strong SEO. Go figure. :)

Posted by Glenn Gabe @ 8:54 am in adwords, google, sem, seo

2 Comments »

Archive for the ‘google’ Category

Robots.txt and Invisible Characters – How One Hidden Character Could Cause SEO Problems

How To Properly Demote Sitelinks in Google Webmaster Tools

Enhanced Campaigns and Local Search | How To Use Bid Adjustments with Radius Targeting to Customize Bids Per Location [Tutorial]

Faulty Redirects, Duplicate Content, and SEO – How a Redirect Glitch Created Hundreds of Thousands of Duplicate Pages

Introducing SEO Bootcamp Princeton, A Hands-On SEO Training Course in Princeton NJ

How Google Analytics Really Handles Referring Traffic Sources [Experiment] – Why Clicks and Visits Might Not Match Up

Hunting False Negatives – How To Avoid False Negatives When Checking Redirects After a Website Redesign or Migration [Screaming Frog Tutorial]

Dynamic Search Ads (DSA) in Google AdWords – How Your Technical SEO Problems Just Impacted Your SEM Campaigns

About G-Squared Interactive

GSQi Links

Follow Glenn Gabe on Twitter

Like GSQi on Facebook

Most Recent Posts

Most Popular Blog Posts

Search GSQi

Archives

More Archives