30 SEO Problems And The Tools To Solve Them

Last November, I authored a popular post on SEOmoz detailing 15 SEO Problems and the Tools to Solve Them. It focused on a number of free tools and SEOmoz PRO tools. Today, I’m finishing up that project with a stab at another set of thorny issues that continually confound SEOs and how some new (and old) tools can come to the rescue.

Some of these are obvious and well known, others are obscure and brand new. All of them solve problems – and that’s why tools should exist in the first place. Below, you’ll find 20+ tools that answer serious issues in smart, powerful ways.

#1 – Generating XML Sitemap Files

The Problem: XML Sitemap files can be challenging to build, particularly as sites scale over a few hundred or few thousand URLs. SEOs need tools to build these, as they can substantively add to a site’s indexation and potential to earn search traffic.

Tools to Solve It: GSiteCrawler, Google Sitemap Generator

GSiteCrawler: Downloadable software to create XML Sitemaps

Download a few files from Google Code and Install on Your Webserver

Looks like Google Webmaster Tools, doesn’t it?

Both GSiteCrawler & Google Sitemap Generator require a bit of technical know-how, but even non-programmers (like me) can stumble their way through and build efficient and effective XML Sitemaps.

#2 – Tracking the Virality of Blog/Feed Content

The Problem: Even experienced bloggers have trouble predicting which posts will “go wide” and which will fall flat. To improve your track record, you need historical data to help show you where and how your posts are performing in the wild world of social media. What’s needed is a cloud based tracking tool that can synch up with the Twitters, Facebooks, Diggs, Reddits, Stumbleupons & Delicious’ of the web to provide these metrics in an easy-to-use, historical view.

Tools to Solve It: PostRank Analytics

PostRank’s nightly emails keep me wracking my brains for better blog post ideas

PostRank sends me nightly reports on how the SEOmoz blog performs across the web – numbers from Digg, Delicious, StumbleUpon, Twitter, Facebook and more. By using this, I can get a rough sense of how posts perform in the social media marketplace and, over time, hopefully train me to author more interesting content.

#3 – Comparing the Relative Traffic Levels of Multiple Sites

The Problem: We all want to know not only how we’re doing with web traffic, but how it compares to the competition. Free services like Compete.com and Alexa have well-documented accuracy problems and paid services like Hitwise, Comscore & Nielsen cost an arm and a leg (and even then, don’t perform particularly well with sites in the sub-million visits/month range).

Tools to Solve It: Quantcast, Google Trends for Websites

If a site has been “Quantified,” no other competitive traffic tool on the web will be as accurate

Since both sites are “Quantified,” I can be sure the data quality is excellent

I’ve complained previously about the inaccuracies of Alexa (as have many others). It’s really for entertainment purposes only. Compete.com is better, but still suffers from lots of inaccuracy, data gaps, directionally wrong estimates and a general feeling of unreliability in the marketplace. Quantcast, on the other hand, is excellent for comparing sites that have entered their “Quantified” program. This involves putting Quantcast’s tracking code onto each page of the site; you’re basically peeking into their analytics.

Sadly, Quantcast isn’t on every site (and their guesstimates appear no better than Compete when they don’t have direct data). Fortunately, one organization has stepped up with a surprisingly good alternative – Google.

Google Trends for Websites allows you to plug in domains and see traffic levels. Much like AdWords Keyword Tool, the numbers themselves seem to run high, but the comparison often looks much better. Google Trends has become the only traffic estimator I trust – still only as far as I could throw a Google Mini, but better than nothing.

#4 – Seeing Pages the Way Search Engine Do

The Problem: Every engineering & development team builds web pages in unique ways. This is great for making the Internet an innovative place, but it can make for nightmares when optimizing for search engines. As professional SEOs, we need to be able to see pages, whether in development environments or live on the web the same way the engines do.

Tools to Solve It: SEO-Browser, Google Cached Snapshot, New Mozbar

_
A longtime favorite site of mine, SEO Browser lets you surf like an engine

_
Poor Google; that’s all they see when they crawl our pretty site

SEO-Browser is a great way to get a quick sense of what the engines can see as they crawl your site’s pages and links. The world of engines may seem a bit drab, but it can also save your hide in the event that you’ve put out code or pages that engines can’t properly parse.

_
I wonder if Googlebot ever gets tired of blue, purple and gray…

Google’s own cached snapshot of a page (available via a search query, as a bookmarklet, or in the mozbar’s dropdown) is the ultimate research tool to know what the engine “sees.” The only trouble is that it works in the past only (and only on pages that allow caching). To get a preview, SEO Browser or our friend below can be useful.

The mozbar lets you dress up like Google whenever the occassion is right

One of Will Critchlow’s feature requests in the new mozbar was the ability to switch user agents, turn off javascript and images and, in essence, become the bot in your browser. Luckily, he also forced us to place a gray overlay in the right-hand corner that alerts you to the settings you’ve changed and gives you an easy, one-click “return to normal.” Browsing like a bot = solved!

#5 – Identifying Crawl Errors

The Problem: Discovering problems on a site like 302 redirects (that should be 301s), pages that are blocked by robots.txt (here’s why that’s a bad idea), missing title tags, duplicate/similar content, 40x and 50x errors, etc. is a task no human can efficiently perform. We need the help of robots – automated crawlers who can dig through a site, find the issues and notify us.

Tools to Solve It: GSiteCrawler, Xenu, GGWMT

Mmmm… Parallel Threads

She canna hold on much longer cap’n!

We’ve already covered GSiteCrawler in this post, but for those unaware, it can be a great diagnostic tool as well as a Sitemap builder. Xenu is much the same, though somewhat more intuitive for this purpose. Tom’s written very elegantly about it in the past, so I won’t rehash much, other than to say – it shows errors & potential issues Google Webmaster Tools doesn’t, and that can be a lifesaver.

Doh! I think we messed up some stuff when KW Difficulty relaunched

Google Webmaster Tools is extremely popular, well known and well used. And yet… lots of us still have crawl errors we haven’t addressed (just look at the 500+ problems on SEOmoz.org in the screenshot above). Exporting to Excel, sorting, and sending to engineering with fixes for each type of issue can save a lot of heartache and earn back a lot of lost traffic and link juice.

#6 – Determine if Links to Your Site Have Been Lost

The Problem: Sites don’t always do a great job maintaining their pages and links (according to our data, 75% of the web disappears in 6 months). Many times, these vanishing pages and links are of great interest to SEOs, who want to know whether their link acquisition and campaigning efforts are being maintained. But how do you confirm if the links to your site that were built last month are still around today?

Tools to Solve It: Virante’s Link Atrophy Diagnosis

Does that mean Stuntdubl & SEOmoz are “going steady?”

This tool comes courtesy of the great team over at Virante, and it’s a pretty terrific application of an SEO need and Linkscape data through the SEOmoz API. The tool will check the links reported from Linkscape/Open Site Explorer and determine which, if any, have been lost. Many times it’s just links off the front page of blogs or news sites as archives fall to the back, but sometimes it can help you ID a link partner or source that’s no longer pointing your way in order to facilitate a quick, painless reclamation. The best part is there’s no registration or installation required – it’s entirely plug and play.

#7 – Find 404 Errors on a Site (without GG WM Tools) and Create 301s

The Problem: Google’s Webmaster Tools are great for spotting 404s, but the data can be, at times, unwieldly (as when thousands of pages are 404ing, but only a few of them really matter) and it’s only available if you can get access to the Webmaster Tools account (which can stymie plenty of SEOs in the marketing department or from external consultancies). We need a tool to help spot those important, highly linked-to 404s and turn them into 301s.

Tools to Solve It: Virante’s PageRank Recovery Tool

3.99 mozRank for ~0.00 effort

The thinking behind this tool is brilliant, because it solves a problem from end to end. By not only grabbing well-linked-to pages that 404, but actually writing the code to create an .htaccess file with 301s to your choice of pages, the tool is a “no-brainer” solution.

#8 - See New Links that are Sending Traffic (and Old Ones that Have Stopped)

The Problem: Most analytics tools have an export function that, combined with some clever Excel, could help you puzzle out the sites/pages that haev started to send you traffic (and those that once were but have stopped). It’s a pain – manual labor, easy to screw up and not a particularly excellent use of your precious time.

Tools to Solve It: Enquisite

I love the ability to look across the past few months and see the trend of new pages and new domains sending links, as well as identifying links that have stopped sending traffic. Some of those may be ripe for reclamation, others might just need a nudge to mention or link over in their next piece/post. This report is also a great way to judge how link building campaigns are performing on the less-SEO focused pivot, sending direct traffic.

#9 – Research Trending/Temporal Popularity of Keywords

The Problem: Keyword demand fluctuates over time, sometimes with little warning. Knowing how search volume is impacted by trending and geography is critical to SEOs targeting fields with these demand fluxes.

Tools to Solve It: Google Insights, Trendistic

Hmmm…. Maybe we should launch Open Webmaster Tools next?

We need to make it out to India & Brazil more often, too!

Google Insights is great for seeing keyword trending, related terms and countries of popularity (though the last of these we’ve found to be somewhat suspect at times). However, sometimes you’re really interested in what’s about to become popular. For that, turning to trend sites can be a big help.

Although it doesn’t yet have a “suggest” feature to help identify terms & phrases that may soon become popular searches, it does help establish the “tipping point” at which a buzzword in Twitter may become a trend in web search. As we’ve discussed in the WhiteBoard Friday on Twitter as an SEO Research Tool, finding the spot at which search volume begins spiking can present big opportunities for fresh content.

#10 – Analyze Domain Ownership & Hosting Data

The Problem: When researching domains to buy, considering partnerships or conducting competitive analysis, data about a site’s hosting and ownership can be essential steps in the process.

Tools to Solve It: Domaintools

We should make sure to re-register this domain…

Long the gold standard in the domainer’s toolbox, DomainTools (once called whois.sc) provides in-depth research about a domain’s owners, their server and, sometimes most interestingly, the other domains owned by that entity. BTW – they’re spot on; SEOmoz owns about 80 other domains besides our own (though we only really use this one and OpenSiteExplorer right now).

#11 – Investigate a Site/Page’s History

The Problem: What happened on this page last month or last year? When conducting web research about links, traffic and content, we all need the ability to go “back in time” and see what had previously existed on our sites/pages (or those of competitors/link sources/etc). Did traffic referrals drop? Have search rankings changed dramatically? Did a previously available piece of content fall off the web? The question really is – how do we answer these questions?

Tools to Solve It: Wayback Machine

Before 2005, we were on a different domain!

_
If you remember this version of the site, you’re officially “old school”

Yeah, yeah, you’ve probably heard of the Wayback Machine, powered by Alexa’s archive of the Internet and endlessly entertaining to web researchers and pranksters alike. What might surprise you is how valuable it can be as an SEO diagnostic tool, particularly when you’re performing an investigation into a site that doesn’t keep good records of its activity. Reversing a penalty, a rankings drop, an oddity in traffic, etc. can consume massive amounts of time if you don’t know where to look and how. Add Wayback to the CSI weapons cache – it will come in handy.

#12 - Determine Semantically Connected Terms/Phrases

The Problem: Chances are, the search engines are doing some form of semantic analysis (looking at the words and phrases on a page around a topic to determine its potential relevance to the query). Thus, employing these “connected” keywords on your pages is a best practice for good SEO (and probably quite helpful to users in many cases as well). The big question is – which words & phrases are related (in the search engines’ eyes) to the ones I’m targeting?

Tools to Solve It: Google Wonder Wheel

_
Nothing about “Yellow Shoes?”

We don’t know for certain that this is a technique that provides massive benefit, but we’re optimistic that tests are going to show it has some value. If you’d like to participate in the experiment, take related phrases from the Wonder Wheel and employ on your pages. Please do report back with details

#13 – Analyze a Page’s Optimization of Images

The Problem: When image search and image accessibility/optimization is critical to your business/client, you need tools to help analyze a page’s consistency and adherence to best practices in handling image dimensions, alt attributes, etc.

Tools to Solve It: Image Analyzer from Juicy Studio

Doh! We need to add some dimensions onto our images.

It’s not the prettiest tool in the world, but it does get the job done. The image analyzer will give any page a thorough evaluation, showing missing alt tags, image dimensions (which can help with page rendering speed) and informing you of the names/alts in a thorough list. If you have image galleries you’re aiming at image search optimization, this is a great diagnostic system.

#14 – Instant Usability Testing

The Problem: Fast feedback on a new landing page, product page, tool design or web page (of any kind) can be essential to smoothing over rough launches. But tools aren’t enough – we need actual human beings (and not the biased ones in our friend groups or company) giving fast, functional feedback. That’s a challenge.

Tools to Solve It: Five Second Test, Feedback Army

It can’t be that easy, can it?

Wow… It totally is! Here I am helping give feedback to a local geek squad.

Users are easier to come by than we think

Both FeedbackArmy & FiveSecondTest offer the remarkable ability to get instant feedback from real users on any page, function or tool you want to test at a fraction of the price normal usability testing requires. What I love is that because it’s so easy, it makes that first, critical step of reaching out to users a low barrier to entry. Over time, I hope systems like these help make the web as a whole a more friendly, easy-to-use experience. Now there’s not excuse!

#15 – Measure Tweet Activity to a URL Across Multiple URL Shortener Platforms

The Problem: You’ve got your bit.ly, your j.mp, your tinyurl, your ow.ly and dozens more URL shorteners. Between this plethora of options and standard HTML links pasted into tweets, keeping up with all the places your URL is being shared can be a big challenge.

Tools to Solve It: Backtweets

Tweeting links in the middle of the night is fun!

Bit.ly can track bit.ly and many other services offer their own tracking systems, but only Backtweets is aggregating all of the sources and making it easy to see what people are saying about your pages no matter how they encode it. Now if only we could get this to integrate with PostRank and Search.Twitter.com and Trendistic and make the interface super-gorgeous and have it integrate with Google Analytics… and… and…

#16 – BONUS: Determining Keyword Competition Levels

Bonus! I mentioned last week in a comment that I’d make a post about the new Keyword Difficulty Tool. Since this post is all about tools anyway, I figured I’d toss it in and save you the trouble of clicking an extra link in your feedreader.

The Problem: Figuring out which keywords have more/less demand than which others is easy (and Google does a great job of it most of the time).

Tools to Solve It: New Keyword Difficulty Tool

The real problem was that our previous keyword difficulty tool attempted to use 2nd order effects and non-direct metrics to estimate the competitiveness level of a particular keyword term/phrase. While it’s true that more popular/searched-for keywords TEND to be more competitive, this is certainly not always the case (and in fact, SEOs probably care a lot more about when a keyword has high traffic and relatively weak sites/pages in the SERPs more than anything else). The new tool attempts to fix this by relying on Page Authority (correlation data here) and using a weighted average of the top ranking sites and pages.

Running five keywords at a time is way better than one
(we’re working to add more – promise)

The best bet here looks like “best running shoes” – relatively lower difficulty, but still high volume

Oh yeah, looking at the top positions, a few dozen good links and some on-page and we’re there

Reversing the rankings is never easy, but parsing through KW Difficulty reports certainly makes it less time-consuming. Watch out for the scores, though – a 65% is pretty darn tough, and even a 40% is no walk in the park. At last, I feel really good about this tool; it was suffering for a good 18 months, and it’s nice to have it back in my primary repertoire with such solid functionality.

Information courtesy of SEOmoz.

Open Site Explorer

For the past 15 months, SEomoz has been working hard to improve Linkscape, their index of the WWW. Today, they are releasing an entirely new platform for Linkscape’s index with more accessible data than ever before. And, for the next 48 hours, full functionality is available entirely for free:

The new tool, Open Site Explorer, makes gathering, sorting and exporting link data easier than ever. It’s built with speed and accessibilty at the forefront and provides a tremendous amount of information about the links to any page or site. Since there’s a lot to cover, let’s dive right into some of the features and functionality.

#1 – Fast Access to Top Level Metrics

At the top of every results page, you’ll find the key metrics we have on your page – the importance/ranking ability of that URL (Page Authority) and root domain (Domain Authority), the number of linking root domains and the total number of links.

#2 – See Up to 10,000 Links Alongside Anchor Text & Key Metrics

You can browse through up to 10,000 links (this is restricted to 1,000 for non-PRO members normally, but will be completely free to everyone for the first 48 hours). We also offer CSV export functionality, but it won’t be available until the weekend (and then, only to PRO members – CSV takes up a LOT of bandwidth for 10K rows ).

#3 – Filtering for the Links You Want to See

As you drill down in the list of links, you can exclude nofollowed links or see only the 301s that point to a page. You also have the ability to sort by the location from which you want to see links – internal vs. external – and links that point to a given page, all pages on a subdomain or an entire root domain.

#4 – Display Root Domains that Contain Links

The second tab in Open Site Explorer (OSE for short) is the linking root domains. We realized that a lot of people want to get a quick glance of the types of sites that are sending links to a given page or domain, and thus created this unique view. In the future (probably a couple months away), you’ll also be able to click an individual domain and see a list of pages from that site that link to the target of your choice.

#5 – Review Anchor Text Term & Phrase Distribution

Anchor text is often the missing link in a “why does that guy rank there?” puzzle. We’re opening up the anchor text distribution so you can learn more about your own sites and pages and those of the competition. You can also sort by both the number of root domains that contain a link with a particular anchor text term (single word) or phrase and the raw number of links containing that anchor text.

#6 – Pie Chart Displays of Link Data

Many SEOs worry that, particularly on small sites, they may be seeing lots of numbers of links, but the sources aren’t ideal. In this view, we try to illustrate through pie charts the percentage of links that come from internal vs. external pages and are followed vs. nofollowed. This view is at the top of the “full metrics” tab.

#7 - Rejoice in Data Junkie Heaven

Additionally in the “full metrics” tab, you’ll find a list of all the Linkscape data we’ve got including mozRank (an algorithm similar to Google’s PageRank), mozTrust (akin to TrustRank) and many more. You can also see the more refined link counts and data for an individual URL, the subdomain it’s on and the hosting root domain.

#8 – Compare Pages/Sites Link Metrics to One Another

A frequently requested feature is the ability to compare one site/page against another. OSE makes this quick and easy with a comparison view drop-down. If you click the “-” symbol again, you can return to the individual report view.

#9 – Graphical Views of Metric Comparisons

In the comparison view, we show nice visual charts that you can embed in a client report or send to your boss to help illustrate just how challenging it might be to take on a particular competitor. For example, you can see above that Fred Wilson has a long way to go to reach Guy Kawasaki‘s stats on his blog (granted, Guy’s posts are designed for a much broader audience and he’s been blogging for longer).

#10 – Compare Links Side by Side

At the bottom of this comparative view you’ll see links side-by-side. We noticed a lot of SEOs open two browser windows with lists of links to compare them against one another and thought “why not make that easier?!” With this feature, you can scroll through the links for two pages to get a fast sense for the quality and variety of sources that point to each.

New Metrics – Domain Authority & Page Authority

We’ve got much more information coming soon about these two metrics, but basically, we’re using our ranking models to build predictions about how well an individual page might perform in the search engines (Page Authority) or how well content on a root domain would do (Domain Authority). These aren’t like PageRank or mozRank at all – they’re much broader.

Authority scores take into account all the metrics we have about a page and hundreds of derivatives of those metrics. We’ve put the scores on a classic 0-100 scale that’s logarithmic (so moving from a 50 to a 60 is much harder than moving from a 10 to a 20). Over time, these metrics will change and evolve as we get better and better with our machine learning systems (and as the engines and the web itself changes). Watch for this week’s Whiteboard Friday with much more detail on this subject. For now Open Site Explorer is the only place to get Domain/Page Authority data, but we’ll be rolling it into the SEOmoz toolbar and other tools over the next few months.

Linkscape’s Index Update

Linkscape itself has also updated – growing to a whopping 65 billion URLs with 45 day minimum freshness. As Nick’s previous post on the Trillion+ URLs Linkscape has seen shows, freshness is one of the most critical metrics for those who care about accurate link data, and we’re working hard to keep our index as up-to-date as possible. Linkscape recrawls every page in the index each month, so no “old data” is stored or served. Our current metrics for this index are:

• Pages: 64,180,990,434 (65 billion)
  • 301s: 293 million
  • 302s: 672 million (Marshall Simmonds calls this “job security”)
  • 404s: 360 million (but we do try to exclude known 404s in crawls, so this may be low percentage wise)
• Subdomains: 259,977,972 (260 million)
• Root Domains: 63,264,651 (63 million)
  • .com – 49.4%
  • .net – 6.4%
  • .de – 5.8%
  • .org – 5.2%
  • .ru – 2.5%
  • .cn – 2.5%
• Links: 701,881,850,733 (701 billion)
  • Nofollows: 13 billion (1.85%)
  • Internal Nofollows: 9.06 billion vs. External Nofollows: 4.11 billion
  • Meta Refreshes: 40.9 million
  • Internal Links: 638 billion vs. External Links: 63 billion (people link to their own stuff a lot more than they do to others)
  • Feed Autodiscovery (i.e. RSS/Atom feeds): 2.261 billion
  • Rel=canonical: 100 million
  • Links passed through 301s: 8.61 billion (just over 1% of all links go through a 301)
• mozRank Correlations to Google Toolbar PageRank
  • Individual page mR: 0.42 (avg. error +/- 0.56 from PR)
  • Subdomain mR: 0.45 (avg. error +/- 0.35 from PR)
  • Root domain mR: 0.45 (avg error +/- 0.37 from PR
• File Extensions
  • html: 26.5%
  • php: 21.7%
  • htm: 10.6%
  • asp: 5.7%
  • aspx: 2.9%
  • cgi: 0.89%

API Update

Finally, we’ve also updated the SEOmoz API – you can now get lists of links for any URL for FREE along with tons of other link data and metrics. Sarah & Nick have a blog post coming soon with more, but for now, check out the API page to get a developer key and the API Wiki for more details.

Answers to Common Questions About OSE

What’s the difference between OSE and Linkscape?

Open Site Explorer provides a fast, free, more basic view of link data while Linkscape provides power users the ability to refine by dozens of filters, search within link anchor text, URLs and domains. Linkscape will let you dig into significantly more metrics and details on a per link basis on things like mozRank passed, Domain mozTrust, juice per anchor text, links from particular TLDs, etc.

OSE is substantively faster than Linkscape, and not as metrics heavy. It’s designed to give the “500 foot view” vs. the deep, in-the-weeds look you can get in Linkscape. Certainly feel free to try both and use the one that suits you best.

Why is OSE on a separate domain?

Three big reasons, actually:

1. We’ve haven’t tried the microsite strategy in a long time (since the first launch of the Web 2.0 Awards), and want to test and see lots of SEO and strategic/branding (we’ll have some cool data to report in the next few weeks/months)
2. OSE is built entirely on the SEOmoz API platform – we wanted to show off just how much you can build using that service
3. SEOmoz engineers are very busy working on another exciting launch (scheduled for June) so we wanted to split resources without putting a load on folks focused on our site (PRO members may see some previews of that even earlier)

What will OSE continue to offer for free?

For the first 48 hours, registered members (anyone with a free SEOmoz account) will get the full PRO features (unlimited metrics, up to 10K links per report, full anchor text data, etc). After that, anyone can still get up to 1,000 links per search and a sampling of metrics. You can see a full breakdown in the bottom right-hand corner of the homepage.

Why Call it “Open” Site Explorer?

We’re aiming to give out more link data than anyone else on the web for free. Open Site Explorer not only gives out lots and lots of links (up to 1,000), but also metrics and link numbers for free (permanently). We also provide a free API that lets you use any of the data (including lists of links) in your applications, public or private. Our goal is to be transparent with this data – to show exactly how many pages/domains are in our index, show accuracy with freshness and canonicalize and re-crawl like a search engine. We’re trying to take the web’s link graph and make it as available as possible and use the revenue component of PRO membership to accelerate growth on index freshness, quality and size.

Please Give Us Feedback!

We’d love to hear from you. If you have suggestions, bug reports (this is a first launch, after all) or ideas for future iterations, please leave them in the comments or send them via the Open Site Explorer feedback form. We’re of course very excited for the launch of OSE and would certainly appreciate you sharing and helping us spread it around. The free period ends at 8am Pacific on Friday, January 22nd, but PRO members will continue to be able to access all the features and unlimited reports (and free reports will still provide up to 1,000 links).

p.s. Two great posts with more information on this topic appeared in the last 24 hours and are worth sharing:

• Open Site Explorer Bookmarklets from MediaWorks UK
• Six Highly Actionable Tips for Open Site Explorer from SEOGadget

Information courtesy of SEOmoz.

Indexation for SEO

How many pages has Google indexed?

This question and the problems surrounding it run rampant through the SEO world. It usually arises when someone starts doing searches like this:

Google claims to have 93,800 pages indexed on the root domain, seomoz.org. That sounds pretty good, but when I ran that search query last week, the number was closer to 75,000 and when I run it again from Google.co.uk 60 seconds later, the number changes even more dramatically:

How about if I hit refresh on my Google.com results again:

Doh! Google just dropped 8,500 of my pages out of their index. That sucks – but not nearly as much as managers, marketing directors and CEOs who use these numbers as actual KPIs! Can you imagine? A number that means nothing, fluctuates 300% between data centers, can change at a moment’s notice and provides no actionable insight being used as a business metric?

And yet… It happens.

Fortunately, there’s an easy way to get much, much better data than what the search engines provide through “site:” queries and this post is here to walk you through that process step-by-step.

Step 1: Go to Traffic Sources in Your Analytics

Click the “traffic sources” link in Google analytics or Omniture (it can also be called “referring sources” in other analytics packages).

Step 2: Head to the Search Engines Section

We want to find out how many pages the search engines have indexed, so the obvious next step is to go to the “search engines” sub-section.

Step 3: Choose an Engine

Choose the engine you want indexation data on and click. If you have both paid and organic traffic from this engine, you’ll want to display organic only at this step, too.

Step 4: Filter by Landing Pages

The “Landing Page” filter in the dropdown will show you the traffic each individual page on your site received from the engine you’ve selected. This also produces the magical “total” number of pages that have received traffic, described in the last step.

Step 5: Record the Number at the Bottom

That count tells you the unique number of pages that received at least one visit from searches performed on Google. It’s the Holy Grail of indexation – a number you can accurately track over time to see how the search engine is indexing your site. On its own, it isn’t particularly useful, but over time (I usually recommend recording monthly, but for some sites, every 2-3 months can make more sense), it gives you insight into whether your pages are doing better or worse at drawing in traffic from the engine.

Now, technically I’m being a bit cheeky here. This number doesn’t tell you the full story – it’s not showing the actual number of pages a search engine has crawled or indexed on your site, but it does tell you the unique number of URLs that received at least 1 visit from the engine. In my opinion this data is far more accurate and more actionable. The first adjective – accurate – is hard to argue (particularly given the visual evidence atop this post), but the second requires a bit of an explanation.

Why is Number of Pages Receiving ≥1 Visit Actionable?

Indexation numbers alone are useless. Businesses and websites use them as KPIs because they want to know if, over time, more of their pages are making their way into the engines’ indices. I’d argue that actually, you don’t care if your pages are in the indices – you care if your pages have the opportunity to EARN TRAFFIC!

Being a row in a search index means nothing if your page is:

• too low in PageRank/link juice to appear in any results
• displaying content the engines can’t properly parse
• devoid of keywords or content that could send traffic
• broken, misdirected or unavailable
• a duplicate of other pages that the engine will rank instead

Thus, the metric you want to count over time isn’t (in most cases) number of pages indexed, it’s number of pages that earned traffic. Over time, that’s the number you want to rise, the number you want marketers to concentrate on and the KPI that’s meaningful. It tells you whether the engine is crawling, indexing AND listing your pages in the results where someone might (has) actually click(ed) them.

If the number drops, you can investigate the actual pages that are no longer receiving traffic by exporting the data to Excel and doing a side-by-side with the previous month. If the number rises, you can see the new pages getting traffic. Those individual URLs will tell a story – of pages that broke, that stopped being linked-to, that fell too far down in paginated results or lost their unique content. It’s so much better than playing the mystery game that SEOs so often confront in the face of “lower indexation numbers” from the site: command.

Some Necessary Caveats

This methodology certainly isn’t perfect, and there are some important points to be aware of (thanks especially to some folks in the comments who brought these up):

• Google Analytics (and many other analytics packages) use sampled data at times to make guesstimates. If you want to be sure you’re getting the absolute best number, export to CSV and do the side-by-side in Excel. You can even expunge similar results from two time period to see only those pages that uniquely did/didn’t receive traffic. In many of these cases, you might also only care about pages that gained/lost 5/10/20+ visits.
• Greater accuracy can be found from shrinking the time period in the analytics, but it also reduces the liklihood that a page receiving very long tail query traffic once in a blue moon will be properly listed, so adjust accordingly, and plan for imperfect data. This method isn’t foolproof, but it is (in my opinion), better than the random roulette wheel of site: queries.
• This technique isn’t going to help you catch other kinds of SEO issues like duplicate content (it can in some cases, but it’s not as good as something like GG WM Tools reporting) or 301s, 302s, etc. which can require a crawling solution.

I’d, of course, love your feedback. I know many SEOs are addicted to and supportive of the site: command numbers as a way to measure progress, so maybe there’s things I’m not considering or situations where it makes sense. I also know that many of you like the number reported in Google Webmaster tools under the Sitemaps crawl data (I’m skeptical of this too, for the record) and I’d like to hear how you find value with that data as well.

p.s. Tomorrow we’ll be announcing two webinars (open to all) about using Open Site Explorer to get ACTIONABLE data. Be sure to leave either Wednesday the 27th at 2pm Pacific or Thursday the 28th at 10am Pacific free

Information courtesy of SEOmoz.

Try SEO Yorkshire and Keep Your Website Updated With Fresh Content

If you have a website, it is important that you don’t keep it in the dark where no one will find it. One of the best ways to help people find your site is to update it with fresh content on a regular basis. By doing this, you will encourage regular visitors and this could turn window-shoppers into genuine sales.

The North Has It Covered

If you opt for SEO, Yorkshire services in the UK are a popular option. Away from the busy Southern cities, Leeds is a hot destination for webmasters looking to find suitable SEO specialists that can help get their site to the top. Plus, if you live in the North of the UK you’ll be able to appreciate that there is a much greater choice for search engine optimisation services locally and there is no need to head down South unnecessarily. In Leeds, for example, an SEO guru will be able to advise you about getting fresh content on your site and the importance of doing so. Here are some things he may tell you:

• Fresh content helps get your site re-indexed by Google
• Updating the site in general will encourage users to return and bookmark your site
• Keep your stock list up to date and remove old items/dead links

Indeed, it is important for any website to look at SEO and the content is always one of the key factors. You should make sure that the content on your site is appropriate. Avoid long blocks of text as this could encourage people to move off the site quickly. Long chunks of text are uninspiring and hard to read. Even if you have some really great information nestled in there, you won’t get any credit for it because people will have clicked off the site before getting to read it. In addition to this, excessively long paragraphs can appear very “spammy” and throw up a red flag to Google, especially if you have tried keyword stuffing too. Many webmasters just starting out in SEO often make these mistakes, and they can always be avoided by just knowing these simple tips.

Trust In The Experts

Of course, it is highly recommended that you speak with a specialist in this area. There are SEO Yorkshire businesses that can help optimise your site and steer you in the right direction. One should never play around with SEO concepts without knowing fully how they can impact long-term on the site, so don’t risk it. Opt for SEO Leeds services today.

SEOmoz’s Ranking Factors 2009

An excellent post by Rand.

If you haven’t checked out SEOmoz before, follow some of the links below and you will soon be hooked. SEOmoz are one of the top SEO agencies on the planet and if you are serious about SEO you should be following them on a regular basis.

I’m thrilled to announce that after months of hard work, SEOmoz’s biennial Search Engine Ranking Factors is finally launching. Every two years, we survey 100 of the industry’s top SEO minds. In 2009, 72 SEOs participated in the data gathering process, answering survey questions that consumed hours of time. The resulting document is an amazing aggregation of data about how search engines rank documents and, at least in my opinion, should be read by anyone serious about practicing search engine optimization.

The document contains five important sections:

• The Overview – offering the most high level view of the ranking elements
• Ranking Factors – the raw data, showing the importance and level of consensus for each factor; this year also includes a set of opinions on how geo-targeting across countries is perceived.
• Link Building – this year, we’ve also added a section asking our SEO participants which methods they find most effective for link acquisition. I think this data is tremendously valuable and interesting for anyone seeking to engage in link building campaigns.
• Additional SEO Data – we asked a few specific questions around SEO to gauge the opinions of the experts; lots of cool stuff in here, too
• Contributors – a list of those who participated in the survey and details about who they are and where you can find them on the web

My great thanks goes out to Timmy & Sam here at SEOmoz, who helped create this year’s document and to all of the generous participants from across the SEO world. Practitioners in more than a dozen countries around the world, all of whom have extremely busy schedules, gave up their time to help those learning SEO get a better view of the subject – please join me in thanking them.

If you’ve got questions, feedback or want to bring up interesting topics, feel free to do so in the comments on this post.

Do you like this post? Yes No

Whiteboard Friday – Matt Cutts on NoFollow

Posted by great scott!

This week we’ve got a special treat! Live from the halls of SES San Jose, our own Jen Lopez sits down with the one-and-only Matt Cutts to discuss NoFollow.

As we all know, there was some controversy about Google’s shifting views on nofollow earlier this year. So now that some time has passed and Big G has refined their position, what would Matt recommend to sites that have lots of nofollow tags already in place? Watch this exclusive interview to find out.

From Jen:

This has been a great week at SES San Jose 2009. There were lots of great sessions, informative tweets, fun swag and I’ve personally met many of our Pro members! My favorite session of the week was “Extreme Makeover: Live Site Clinic” where Matt Cutts, Greg Boser, Elisabeth Osmeloski, Tiffany Lane and Vanessa Fox (unofficially ) reviewed several websites in front of hundreds of people. The use of the rel=nofollow for PR sculpting came up in the review (imagine that). Matt Cutts recommended to a site owner that he remove all the nofollows from his site, even to non-necessary pages.

This is a subject that comes up time and time again on the SEOmoz site whether it’s in the Q & A section, or in the blog and comments. So I wanted to find out directly from Matt, what he would recommend to our users moving forward. This afternoon I sat with Matt and got his take on the conference in general as well as the use of rel=nofollow and PR sculpting. (Oh yea! And if you haven’t seen the tweets and read the posts yet, Matt got a new haircut.

SEOmoz Whiteboard Friday – Matt Cutts on NoFollow from Scott Willoughby on Vimeo.

Do you like this post? Yes No

An SEO’s Dream Come True

Well, it’s this SEO’s dream anyway.

You see, my biggest frustration in life is lack of time. It limits the potential of any given hour or day. I sometimes think to myself – sure, I could be running an SEO analysis on this site… OR I could be sitting next to my wife on the couch, eating chocolate and watching Venture Brothers re-runs. You can see my dilemma.

Enter the new mozbar:
_

Sweet, eh? Download it here.

_
It probably comes as no surprise that, just like nearly every other SEO in the field, I waste a lot of time doing the same tasks over and over again. Namely:

• Looking up the link data of a site/page to get an idea of its relative importance from a link popularity perspective
• Viewing the source code of a page to analyze the meta data, on-page content, and SEO-specific tags

And now, with the new mozbar, I’m spending 1/10th of the time it took me previously to get that critical information.

I’ll show you what I mean. Let’s say I needed to analyze my new friend Chadd’s website. Chadd sells straight edge & safety razors, but I’m not convinced he’s doing all the right things from an SEO perspective:
_

Sharp razors, but is the SEO sharp?

_
Rather than going through the time consuming act of scrolling through Chadd’s nasty source code to find the answers, I can use the mozbar’s nifty “analyze page” feature, thusly:
_

Screw “view source” – this rules!

_
I can clearly see that my friend Chadd needs to do some work if he wants to rank “safety razor.” This view shows me data on:

• The URL
• Page Title
• Meta Description
• Meta Keywords
• H1
• H2
• HTML Text
• Bold/Strong
• Italic/Em
• Alt Text
• Meta Robots
• Rel=”Canonical” usage
• IP Address
• mozRank (for the URL, subdomain and root domain)
• mozTrust (for the URL, subdomain and root domain)
• # of External Links (for the URL, subdomain and root domain)
• # of Linking Domains (for the URL, subdomain and root domain)

Unfortunately, we have lost one feature – Google’s PageRank. The powers that be at Google have asked us to remove that functionality from our toolbar and we’ve complied. We know this is frustrating, but we also need to respect the engines’ wishes – after all, we wouldn’t exist without them If you’re searching for alternatives, I’d suggest SearchStatus (which has a number of cool functions) or Live PageRank (which will give you just the PR bar and you can drag it to whatever location you’d like).

The new mozbar is just launching and will likely have a few bugs – please report these in the comments below and feel free to suggest any additional functionality you’d like to see. We’ll be spec’ing fixes & upgrades in the next couple weeks

Change your life. Download the mozbar.

_
p.s. Last time Chadd (from RetroRazor.com) and I were out together at a party, he gave me a free Retro Razor to try. Does that mean I’m obligated to put a nofollow on the link above, even though I really like the shave?

Do you like this post? Yes No

SEO Yorkshire

SEO Yorkshire offers expert professional help for your business via online promotion of your company’s website, on Search Engines, News Websites and Social Media / Web 2.0 platforms.

Services can be offered in the following areas:

• SEO
• Social Media
• Link Building
• Blogging / RSS Management & Promotion
• Press Releases
• Article Marketing
• Social Networking
• Local Search
• Pay Per Click
• Brand Reputation Management

Please see our “Services Page” for further information.