Web Site Not Indexed by Search Engines?
Some common reasons for search engines to not index your web site ....
INDEX TIME: First, make sure you've allowed enough time to become indexed. The amount of time to allow is sometimes listed on the search engine's submission page. Unfortunately, the engine's own advertised times are often inaccurate or out of date. Average index times often range from one to eight weeks or more depending on the engine. Some engines like AltaVista and Inktomi offer paid options if you wish to be indexed more quickly.
ALREADY INDEXED: Be sure you're not already indexed but just don't know it.
In addition, you cannot simply do a search on a keyword that applies to your Web site and expect it to pop up at the top. In fact, you must take pro-active steps to optimize your pages for each search engine. If you don't, it's very unlikely you'll find your Web site.
The method to determine if a page or domain has been indexed varies from one engine to another, and in many cases, it's difficult to tell for sure whether your pages are in fact in there. Never assume that you're not indexed just because you searched for a bunch of keywords and you never came up in the first few pages of results. You could be there (i.e., indexed) but be buried near the bottom.
MISSING PAGE: Make sure you have uploaded the pages to your site before submitting them. This one will seem obvious to many people, but submitting a page that does not exist or submitting with a subtle typo in the URL is a goof we might all make at one time or another. This is important since not all search engines will notify you if the URL does not exist when you submit.
ROADMAP FROM HOME PAGE: Some engines have been known to drop pages that cannot be traveled to from the home page. You may want to consider submitting your home page that links either directly or indirectly to your doorway pages. Think of your Web site as a series of roads (i.e., links) from one page to another. If there's no road from your home page to the page you want indexed, a search engine may decide the page is unimportant or of low-quality. You could submit the page directly, but the engine may reject it or may drop it at a later date when it finds no "road."
EXTERNAL LINKS: Some search engines have been known to refuse to index Web sites that do not have any other Web sites linking to them. Or, they may index your home page but refuse to index any other pages until you achieve at least one or more links from another domain. Or, they may index you for awhile but then "prune" their database later of all Web sites that did not achieve any external links within a certain period of time.
FRAMES: If you have content inside HTML frames, this can cause problems with submissions. For example, the search engine may index the main content of the page, but not the surrounding menu frame. Visitors to your site will then find some information but may not see the associated menu! It's generally best if you can create non-framed versions of your pages. You should then submit the non-frames versions of your pages, which can of course link to your framed Web site. Alternatively, you can enter your relevant text within the NOFRAMES area of a framed page that most search engine spiders will read. However, don't expect to achieve high rankings while optimizing the NOFRAMES area. Optimizing a NON-framed page will often achieve better results.
SPIDER BLOCKS: Search engine spiders cannot index sites that require any kind of registration or password. A spider cannot fill out a form of any kind. The same rule applies regarding indexing of content from a searchable database. That's because the spider cannot fill out a form to query that database. The solution is to create static pages that the engines will be able to find and index without performing a special action on your site.
FREE SITES: Many engines no longer index pages from free web sites or they limit the number of pages they will index from these hosts. Sometimes they will get too many "junk" submissions from free web site domains such as Geocities or others. Therefore, some engines choose not to index anyone from some of these domains. Or, more commonly, they limit the number of pages they will accept.
It's always best to buy your own domain name (very important) and place it on a respected, paid service to avoid being discriminated against. The free traffic you can generate from the search engines is just too valuable to be sacrificed for the small savings a free hosting service provides. In addition, free hosts are often unreliable or force you to display banners that send valuable visitors away from your Web site soon after arriving. That can cost you sales.
GUILT THROUGH ASSOCIATION: If your Web site shares the same IP address as many other Web sites on your host's Web server, then you may find your IP quietly banned from something another Web site on the same server did! We've heard from many people who tried everything to be indexed only to find it was a snap once they changed hosting services.
SUBMISSION LIMITS: Make sure you're submitting within the recommended limits. Some engines do not like more than a certain number of submissions for the same domain. If you exceed the limit, you may find that all your submissions are ignored. Some submission consultants feel it is dangerous to submit more than ONE page to an engine for a given Web site.
NON-INDEXABLE CONTENT: It's important to know the types of content that the average search engine cannot index. Most engines cannot index text that is embedded in images. Text that appears in multimedia files (audio and video) will not be indexed. Most engines cannot index information that is generated by Java applets, Flash files or in XML coding.
LARGE PAGES: If your site has a slow connection or the pages are very complex and take a long time to load, it might time out before the spider can index all the text. For the benefit of your visitors and the search engines, limit your page size to 50K or less. In fact, most Webmasters recommend that your page size PLUS the size of all your images on the page should not exceed 50K-70K total.
DEEP LINKS: If you submit just your home page, don't expect a search engine to travel more than one or two links away from the home page or from the page that you submitted. Over time they may venture deeper into your site, but don't count on it happening quickly. You'll often need to submit pages individually that appear further down into your site or create more direct links from the home page (either visible links or hidden links). This way the search engines can find them.
* many engines assign "bonus points" to pages they find on their own versus pages that were specifically submitted to them.
UNRELIABLE HOSTS: If your Web site fails to respond when the search engine spider pays a visit, you will not be indexed. Even worse, if you are indexed and they pay a visit when your site is down, you could be removed from their database! Consequently, it pays to have a reliable hosting service that is up 99.5% of the time. However, at some point a spider is going to hit that other 0.5% and end up yanking your pages by mistake. Therefore, it pays to keep a close eye on your listings.
SPAM: If you have ever used any questionable techniques that might be considered an overt attempt at spamming (i.e., excessive repetition of keywords, same color text as background, or other things), an engine may ignore or reject your submissions. If you're having trouble getting indexed in the expected amount of time, make sure your site is spam-free.
REDIRECTS: If your site contains redirects or meta refresh tags these things can sometimes cause the engines to have trouble indexing your site. Generally they will index the page that it is redirecting TO, but if it thinks you are trying to "trick" the engine by using "cloaking" or IP redirection technology that it can detect, there is a chance that it may not index the site at all.
PROPER DIRECTORY SUBMISSIONS: If you're submitting to a directory site like Yahoo, Open Directory, Looksmart, or others, then a human being will review your site. They must decide if the site is of sufficient "quality" before they will list it. Getting listed in major directories first can help you get listed elsewhere.
INDEX TIMES CAN FLUCTUATE: Sometimes engines will index sites every 30 days fairly consistently and then suddenly stop indexing most sites for several months. This can be frustrating, but it does happen. Generally a major engine will not go more than three to four months without refreshing its index. If you're wondering if others are experiencing trouble getting indexed on a particular engine, try asking around.
PAGE LIMITS: If you have many pages indexed but are having a hard time getting new ones recognized, be aware that there are limits. Each search engine will only spider so many pages of your Web site. This may range from a few dozen or three or four hundred depending on the engine. Some people have even been successful in getting far more pages indexed depending on the engine. Google is one engine that tends to crawl deeper into your site. However, how deep they go may depend on factors like your link popularity. Sites with higher link popularity are deemed "worthier" of more thorough indexing.
RANDOM ERRORS: Sometimes the engines just lose submissions at random through technical errors and bugs. After all, they are managing a database of hundreds of millions of pages.
TIP: Once your page achieves a desirable ranking, it's best not to continue submitting it.