Google Launches Program To Help Web Sites Get Indexed

Google today announced its latest salvo in the hot competition in the search landscape: a free online service intended to allow webmasters to automatically submit their Web pages to Google to be indexed in a bid to improve the results of online searches.

Google Sitemaps is a beta release and is designed to be a collaborative program between Google and webmasters intended to provide the search company with more information about Web content.

In an interview with SearchEngineWatch published in conjunction with the launch, Google Engineering Director Shiva Shivakumar explained that “currently Web crawling [searching] is limited. Crawlers don’t know all the pages at a Web site (e.g., dynamic pages), when those pages change, how often to recrawl pages, how much load to put on a Web site. So they try to guess.

“We want to work collaboratively with webmasters to get a big picture of all the URLs we should be crawling, and how often they should be recrawled.”

No Effect on Ranking

The program does not replace the crawl technology Google currently uses to find Web content; it is designed to enhance it. Also, using Sitemaps will neither guarantee that a Web site will be included in the index nor will it have any effect on how sites are ranked on results pages, Google said.

The new offering is the latest in a long line of product releases in recent months in the heated battle for consumers by the various online search companies, from Yahoo’s Mindset service allowing users to control the search sort to Ask Jeeves’ Zoom and Web Answers features aimed at providing more accurate searches to MSN’s Virtual Earth for local search and Google’s own Desktop Search tool and its satellite maps feature.

Google said the Sitemaps launch demonstrates its ongoing efforts to create innovative Web search technologies that make finding relevant information faster and easier.

Until now, webmasters could only publish their pages to the Internet and simply wait for Google to crawl their site for inclusion in the Google search index, or they could submit their home page to Google’s “Add URL” page and hope the search engine found all the pages linked from it.

Technology Will Be Shared

With Google Sitemaps, webmasters can inform Google about all their existing Web pages, prioritize the pages they want crawled first, and tell the search engine company when pages are updated so that Google can index new content faster.

In a blog on the Google Web site, Shivakumar wrote: “We’re undertaking an experiment called Google Sitemaps that will either fail miserably, or succeed beyond our wildest dreams, in making the Web better for webmasters and users alike. It’s a beta ‘ecosystem’ that may help webmasters with two current challenges: keeping Google informed about all of your new Web pages or updates, and increasing the coverage of your Web pages in the Google index.”

According to the blog entry, Google will be sharing the technology so other search sites can improve their service as well. “This project doesn’t just pertain to Google, either: we’re releasing it under the Attribution/Share Alike Creative Commons license so that other search engines can do a better job as well. Eventually we hope this will be supported natively in webservers (e.g. Apache, Lotus Notes, IIS),” Shivakumar wrote.

Google said the new service is intended for all Web sites, from those containing only a single Web page to companies with millions of ever-changing pages.

How It Works

As for whether some Web sites might try to use the new service to spam the search index in bulk, Shivakumar told SearchEngineWatch, “We are always developing new techniques to manage index spam. All those techniques will continue to apply with the Google Sitemaps.”

Here’s how Sitemap works: Webmasters can sign up for the program at www.google.com/webmasters/sitemaps. They then generate and submit an XML formatted site list. This file can be created using the Sitemap Generator, a free open-source tool that generates an XML sitemap for a few simple use cases.

Webmasters do not need a Google account to generate and submit sitemaps. However, if they do sign up, they can log in to check the status of their sitemaps and view diagnostic information for their submissions.

The product is available in the U.S. English and German language interfaces today. Google said it hopes to expand its availability to other languages in the near future.