We’ve recently discussed several ways of handling duplicate content on a single website; today we’ll look at ways of handling similar duplication across different websites, across different domains. For some sites, there are legitimate reasons to duplicate content across different websites — for instance, to migrate to a new domain name using a web server that cannot create server-side redirects. To help with issues that arise on such sites, we’re announcing our support of the cross-domain rel=”canonical” link element.
Ways of handling cross-domain content duplication:
- Choose your preferred domain
When confronted with duplicate content, search engines will generally take one version and filter the others out. This can also happen when multiple domain names are involved, so while search engines are generally pretty good at choosing something reasonable, many webmasters prefer to make that decision themselves.
- Enable crawling and use 301 (permanent) redirects where possible
Where possible, the most important step is often to use appropriate 301 redirects. These redirects send visitors and search engine crawlers to your preferred domain and make it very clear which URL should be indexed. This is generally the preferred method as it gives clear guidance to everyone who accesses the content. Keep in mind that in order for search engine crawlers to discover these redirects, none of the URLs in the redirect chain can be disallowed via a robots.txt file. Don’t forget to handle your www / non-www preference with appropriate redirects and in Webmaster Tools.
- Use the cross-domain rel=”canonical” link element
There are situations where it’s not easily possible to set up redirects. This could be the case when you need to move your website from a server that does not feature server-side redirects. In a situation like this, you can use the rel=”canonical” link element across domains to specify the exact URL of whichever domain is preferred for indexing. While the rel=”canonical” link element is seen as a hint and not an absolute directive, we do try to follow it where possible.
Still have questions?
Q: Do the pages have to be identical?
A: No, but they should be similar. Slight differences are fine.
Q: For technical reasons I can’t include a 1:1 mapping for the URLs on my sites. Can I just point the rel=”canonical” at the homepage of my preferred site?
A: No; this could result in problems. A mapping from old URL to new URL for each URL on the old site is the best way to use rel=”canonical”.
Q: I’m offering my content / product descriptions for syndication. Do my publishers need to use rel=”canonical”?
A: We leave this up to you and your publishers. If the content is similar enough, it might make sense to use rel=”canonical”, if both parties agree.
Q: My server can’t do a 301 (permanent) redirect. Can I use rel=”canonical” to move my site?
A: If it’s at all possible, you should work with your webhost or web server to do a 301 redirect. Keep in mind that we treat rel=”canonical” as a hint, and other search engines may handle it differently. But if a 301 redirect is impossible for some reason, then a rel=”canonical” may work for you. For more information, see our guidelines on moving your site.
Q: Should I use a noindex robots meta tag on pages with a rel=”canonical” link element?
A: No, since those pages would not be equivalent with regards to indexing – one would be allowed while the other would be blocked. Additionally, it’s important that these pages are not disallowed from crawling through a robots.txt file, otherwise search engine crawlers will not be able to discover the rel=”canonical” link element.
We hope this makes it easier for you to handle duplicate content in a user-friendly way. Are there still places where you feel that duplicate content is causing your sites problems? Let us know in the Webmaster Help Forum!
Posted by John Mueller, Webmaster Trends Analyst, Google Zürich
Let’s take a quick look at the individual sections in the Google Webmaster Tools’ Site Performance feature:
Performance overview
The performance overview shows a graph of the aggregated speed numbers for the website, based on the pages that were most frequently accessed by visitors who use the Google Toolbar with the PageRank feature activated. By using data from Google Toolbar users, you don’t have to worry about us testing your site from a location that your users do not use. For example, if your site is in Germany and all your users are in Germany, the chart will reflect the load time as seen in Germany. Similarly, if your users mostly use dial-up connections (or high-speed broadband), that would be reflected in these numbers as well. If only a few visitors of your site use the Google Toolbar, we may not be able to show this data in Webmaster Tools.
The line between the red and the green sections on the chart is the 20th percentile — only 20% of the sites we check are faster than this. This website is pretty close to the 20% mark, which pages would we have to work on first?
Example pages with load times
In this section you can find some example pages along with the average, aggregated load times that users observed while they were on your website. These numbers may differ from what you see as they can come from a variety of different browsers, internet connections and locations. This list can help you to recognize pages which take longer than average to load — pages that slow your users down.
As the page load times are based on actual accesses made by your users, it’s possible that it includes pages which are disallowed from crawling. While Googlebot will not be able to crawl disallowed pages, they may be a significant part of your site’s user experience.
Keep in mind that you may see occasional spikes here, so it’s recommended that you watch the load times over a short period to see what’s stable. If you consistently see very large load times, that probably means that most of your users are seeing very slow page loads (whether due to slow connections or otherwise), so it’s something you should take seriously.
Page Speed suggestions
These suggestions are based on the Page Speed Firefox / Firebug plugin. In order to find the details for these sample URLs, we fetch the page and all its embedded resources with Googlebot. If we are not able to fetch all of embedded content with Googlebot, we may not be able to provide a complete analysis. Similarly, if the servers return slightly modified content for Googlebot than they would for normal users, this may affect what is shown here. For example, some servers return uncompressed content for Googlebot, similar to what would be served to older browsers that do not support gzip-compressed embedded content (this is currently the case for Google Analytics’ “ga.js”).
When looking at flagged issues regarding common third-party code such as website analytics scripts, one factor that can also play a role is how wide-spread these scripts are on the web. If they are common across the web, chances are that the average user’s browser will have already cached the DNS lookup and the content of the script. While these scripts will still be flagged as separate DNS lookups, in practice they might not play a strong role in the actual load time.
We offer these suggestions as a useful guideline regarding possible first performance improvement steps and recommend using the Page Speed plugin (or a similar tool) directly when working on your website. This allows you to better recognize the blocking issues and makes it easy to see how modifications on the server affect the total load time.
For questions about Webmaster Tools and this new feature, feel free to read the Help Center article, search and post in the Webmaster Help Forums or in the Page Speed discussion group. We hope this information helps you make your website even faster!
Posted by John Mueller, Webmaster Trends Analyst, Google Zürich
Google just launched Site Performance, an experimental feature in Webmaster Tools that shows you information about the speed of your site and suggestions for making it faster.
This is a small step in our larger effort to make the web faster. Studies have repeatedly shown that speeding up your site leads to increased user retention and activity, higher revenue and lower costs. Towards the goal of making every webpage load as fast as flipping the pages of a magazine, we have provided articles on best practices, active discussion forums and many tools to diagnose and fix speed issues.
Now we bring data and statistics specifically applicable to your site. On Site Performance, you’ll find how fast your pages load, how they’ve fared over time, how your site’s load time compares to that of other sites, examples of specific pages and their actual page load times, and Page Speed suggestions that can help reduce user-perceived latency. Our goal is to bring you specific and actionable speed information backed by data, so stay tuned for more of this in the future.
The load time data is derived from aggregated information sent by users of your site who have installed the Google Toolbar and opted-in to its enhanced features. We only show the performance charts and tables when there’s enough data, so not all of them may be shown if your site has little traffic. The data currently represents a global average; a specific user may experience your site faster or slower than the average depending on their location and network conditions.
This is a Labs product that is still in development. We hope you find it useful. Please let us know your feedback through the Webmaster Tools Forum.
Update on 12/04/2009: Google’s team just reconvened to provide you more information on this feature. Check out JohnMu’s latest post on Site Performance!
Posted by Sreeram Ramachandran, Software Engineer & Arvind Jain, Director, Faster Web program
