Cookies
Cookies are another way sites preserve state on the Internet. A cookie is a small file stored by the visitor’s browser and read by the site so the site can remember who the visitor is. In general, using cookies for state control works well for site visitors using commercially available browsers; however, this breaks down for the search engine spiders.
Search engine spiders do not accept, store, or transmit cookie information. When a cookie is not stored by the visitor to a site, the following can happen:
• Nothing – the site carries on working. This is great for the search engine spiders, but means that for visitors who decide not to accept cookies, the web site cannot keep state and so things break, such as the shopping cart losing anything placed in it.
• Some web site software will detect if the user can accept cookies, and if they can't, they will use a session ID in the URL to preserve state. This works fine for visitors with cookies turned off; however, the search engines will not index pages with session IDs (see the section on dynamic URLs for details).
The problem persists as most web site owners have cookies turned on and are totally oblivious to the fact that search engines spiders cannot access the site. To test whether you have an issue, disable cookies on your browser and see what the URL looks like. Additionally, spider emulators are available and they should highlight any cookie issues you may have.
Another common problem is when developers require visitors to accept cookies to enter the site. In this case, the spiders cannot enter the site or follow alternative navigation, preventing the site's inclusion.