An Eye on Third-Generation Search

Flash back to the spring of 1995. That’s when Digital Equipment’s Palo Alto Research Lab scientists developed the first searchable full-text database of the World Wide Web. It took the Web by storm later that year as AltaVista, search engine extraordinaire. “AltaVista” means “a view from above,” signifying “big ideas and a fascination with keeping track of information,” according to the site’s “About” page Since those heady days, a lot has changed. It was first acquired (with Digital Equipment) by Compaq Computer in 1998, then by CMGI in 1999.

International Prowess

AltaVista showed an early interest in serving the international community. “We provided the first multilingual search capabilities on the Web in 1997,” said AltaVista Product Marketing VP Chris Kermoian. “Right after that, we introduced Babel Fish, a translation service that includes Chinese, Japanese, and Korean translations.”

Starting in 1999, AltaVista launched local international sites in Germany, Sweden, the U.K., France, Holland, Italy, and Denmark, to name a few. Today, it maintains over 20 country-specific indices, including its main index. More than half of AltaVista’s Web traffic originates outside the United States, making it an excellent vehicle for international visibility. It performs 50 million search queries per day in over 25 languages.

Multimedia Prescience

“We were also one of the first to develop multimedia search technology,” said Kermoian. “AltaVista introduced new image, audio, and video search centers in early 2000, creating the most extensive multimedia library on the Web at the time.” Today, over 20 file types are indexed in various AltaVista collections (most are the various multimedia types: MP3s, JPGs, GIFs, Real Audio or Video, Microsoft Windows Media, Apple QuickTime, etc.).

Diversified Revenue Model

The key to AltaVista’s survival through last year’s downturn was its focus on its core capability of providing Internet search services and enterprise search software. AltaVista shifted from a full-service portal back to its pure search roots, further diversifying its business-to-business (B2B) revenue model by entering the highly promising market of enterprise search software.

“Like many others in the industry,” said Kermoian, “AltaVista has cut costs, streamlining its operations to right-size the business.”

Business units include the Web-based consumer search engine, Internet search services (Web-wide search capabilities for Internet sites), paid-inclusion programs (Trusted Feed, Express Inclusion, Listing Enhancements, etc.), various advertising programs, and its highly successful software division.

The software division provides enterprise-grade search software to clients such as Amazon.com, Borders, the FBI, and NASA, enabling these organizations to turn information into power, cataloging, and rapid data access across organizational networks (intranets, extranets, databases, and customer-facing Web sites). “The growth of AltaVista’s search software division has been a significant revenue source over the past two years,” said Kermoian. “We have licensed information-access and data-retrieval software to more than 1,200 leading companies.”

AltaVista’s paid-inclusion programs also help boost revenues because they must be renewed every six months. Express Inclusion lets you add up to 500 pages at $39 for the first URL (less for subsequent pages). It includes weekly updates for freshness. Trusted Feed is for sites of more than 500 pages and is based on cost-per-click pricing. It also includes weekly updates, and you can submit custom titles, keywords, and abstracts. Both of these programs index dynamic content, giving ad customers deep visibility and users deep Web content.

What Makes AltaVista Tick?

“In the past year, we have doubled the size of our index,” said Kermoian. “We’ve improved our news search and launched a comparison shopping search.” AltaVista has also expanded search results to include additional digital resources and data such as white and yellow pages listings, blended stock quotes, images, maps, and multimedia files.

“Freshness is a major priority,” said Kermoian. “In the last three weeks alone, we’ve added or refreshed over 270 million pages in our index, not including those URLs from our paid-inclusion programs, which are refreshed weekly. In addition, free submissions of new URLs are added on a continual basis, multimedia updates come in daily, and we make 15-minute updates to our news index,” he added.

AltaVista crawled and evaluated 4 billion pages for its current full-page index of 700 million pages. Below is a breakdown on freshness:

How does AltaVista plan to improve the user experience? “We continue to refine our relevancy algorithm, increase the size of our database, and improve our user interface,” said Kermoian. “Going forward, we envision truly user-centric Internet-search and enterprise-grade-information access and data-retrieval technologies, or third-generation search.”

What Is Third-Generation Search?

“AltaVista’s goal is to deliver best-of-breed third-generation search,” explained Kermoian. “First-generation search basically ranked sites based on page content. Second-generation search focused on link analysis, taking the structure of the Web into account. Third-generation search will go beyond the Web, taking additional factors into account, such as time of day, geography, previous searches, stated preferences, and so forth.”

Want High Rankings?

What does it take to get top listings on AltaVista? “Your best bet is to ensure that your content is valuable and properly presented,” said Kermoian. “Developing proper meta-tags is important.” AltaVista uses titles, descriptions, and keywords in its ranking algorithm. “Choosing keywords that target specific queries related to page content is effective,” continued Kermoian, “for instance, if a client’s site is about golf, the meta-tags and site content should reflect this.”

“Avoid spamming or your site may be removed from our index,” he warned. “Sites created specifically for search engine robots should be avoided. AltaVista will index doorway pages only if used as a navigational aid to the rest of the site (e.g., BMW.com) and will not condone doorway spam pages.” He cited a good rule of thumb: “If a site has one doorway page that helps direct users to quality content within, that’s acceptable. But creating multiple doorway pages that send the user to the exact same site is clearly spamming.”

The Role of XML

Like others we’ve asked, AltaVista does not think XML will take over in the near term (four to five years) because of the prevalence of HTML. “If XML becomes pervasive, AltaVista is prepared,” asserts Kermoian. “We currently use XML extensively in our Internet Search Services program (providing search capabilities for other Internet sites) and the Trusted Feed program (receiving representations of Web pages from paid-inclusion customers), as well as many data interchanges with partners.”

Will AltaVista regain its glitter? Only time will tell. It doesn’t have the traffic it used to command, but it’s still a top choice for many serious researchers and is often cited as a source by librarians. Stay tuned — AltaVista plans to make an announcement on February 19.

Defining Search Technology

Most people like Google because it’s easy to use, it’s fast, it has a huge database, and — most important — it works. Remember when Google hit the scene? It was 1998. Stanford computer science Ph.D. candidates Sergey Brin and Larry Page were working on a class project to identify meaningful patterns in Web link structure. They became fascinated with analyzing “backlinks” (pages linked back to a site) and realized these backlinks could help build a better mousetrap.

What’s in a Name?

“Google” is a play on the word “googol,” which was coined by Milton Sirotta, nephew of American mathematician Edward Kasner, to refer to the number represented by one followed by 100 zeros. Google’s use of the term reflects the company’s mission to organize the immense amount of information available on the Web. When Brin and Page presented their idea to the first angel investor, this investor wrote the check to “Google Inc.” After thinking about it for weeks, they figured they’d better open an account in the name of Google Inc. to be able to cash the check. So the legend goes…

The PageRank Phenom

It’s been said that Google changed the face of Internet search with an algorithm known as PageRank. The PageRank algorithm was definitely a technological breakthrough, as most major search engines now use link popularity as part of their relevancy algorithm. So how does it work?

“Google’s PageRank search technology works by first identifying the link structure of the entire Web, then ranking individual pages based on the number and importance of pages linked to them,” said Google software engineer Matt Cutts. My perception when talking to Cutts was that importance (the popularity and relevance of the backlink) counts more than the number of backlinks.

Is There a Weak Spot?

If any, it is that Google works better on searches for specific information (such as “rainfall in Hong Kong”) than for general information (such as “Bible”), because search results aren’t categorized, making the results a bit unwieldy for broad search terms. The Google directory helps, as the directory results appear above all search results.

Newer search tools, such as AllTheWeb, Teoma, and WiseNut, classify their results by category. For instance, Teoma divides search results for “Bible” into the following folders: Bible Study; King James Version; Holy Bible; Virginia Textbook; Bible Prophecy; Versions, Search; Biblical Resource; and First Letter. Most searchers, as a rule, don’t narrow down their queries properly because they’re not used to conducting research.

Can you get greater relevance by categorizing results, and, if so, will Google follow the trend toward categorization? “Google is in its second generation of experimenting with category-based results,” explained Cutts. “Users apparently do not like having too many category options, but presenting clear and concise categories is important to users.”

The Road to Success

Google achieved its success and profitability through two sources of revenue: advertising and search services. The AdWords program is targeted and effective, currently yielding up to five times the average click-through rate (CTR) for traditional banner ads. Cutts reiterated the Google mantra: We do not offer paid inclusion.

For additional revenue, Google provides search services to major Web portals and corporate Web sites. It has over 130 customers in more than 30 countries. These customers include Yahoo and its international properties, Sony and its global affiliates, AOL/Netscape, Cisco Systems, and others. These partners pay Google an upfront search service fee and per 1,000 results delivered to power search on their respective portals or corporate Web sites. For every search conducted on partner sites, Google receives a fee.

The Enhanced Google Toolbar

Since Google released a beta version a few months ago, several million people have downloaded the Google Toolbar. The toolbar allows users to vote on site popularity. This could give Google a reading on site popularity based on opinion rather than link structure alone. However, selective bias is a problem.

You can download the beta version, which allows you to rank search results with a voting button. When asked about incorporating this info into the algorithm, Cutts said, “Rather than using the votes to tinker with the specific rankings of particular pages or sites, the feature would most likely be used to bolster the relevance of overall results.” Cutts indicated that data collected so far is promising, but it would take months before the preliminary data could be of conclusive value.

How Does Google Rank Web Sites?

Basically, it ranks sites by the words listed on each page and the key phrases used in the page’s title and description. The spider looks at about 25 factors, including the keyword and description meta tags. It also ranks the page’s popularity, which is determined by the number and importance of sites linked to the page.

When asked how to gain high rankings, Cutts replied, “The guidelines are pretty simple: Stay away from hidden text, hidden links, cloaking, sneaky redirects, lots of duplicate content on different domains, and doorway pages. Webmasters should also stay away from programs that send automatic queries to Google. The worst thing you can do is try to cheat: Shortcuts to boost PageRank or rankings usually do more harm than good. Even if an SEO [search engine optimizer] does think he’s found a shortcut, about two-thirds of the time it may be a sting operation. Don’t bother with link exchanges, signing guest books, or other tricks — the best use of a Webmaster’s time is building good content — and honestly promoting their [sic] site. When Google punishes spam like cloaking, we sometimes take out not only the cloaked domain but the SEO’s client as well.”

A Look Into the Future

Google is working toward providing a deeper, fresher, and more personalized index. “The future will be about features and more about the overall usefulness of an engine,” said Cutts. “We believe users want relevancy, but they also want quick, clean results with proven integrity,” he added. When asked about XML, Cutts replied, “Not any time soon. The main benefit of HTML is that anyone can write it. That’s part of why the Web had such meteoric growth. XML is great for machine-to-machine communication, but it’s much more difficult for a person to produce by hand.”

During the coming year, Google hopes to increase its lead across the board. “We’ll be introducing new ways to search. We don’t want to give away any secrets, but Google will provide many helpful surprises in 2002,” volunteered Cutts. I understand the company’s focus will be on search and the user experience.

How Deep?

Google does its share of indexing the deep Web by rolling out support for hundreds of file formats found there: PDF, RTF, PostScript, Word, Excel, PowerPoint, and more. It crawls millions of dynamic pages. Google indexes 3 billion Web documents every 28 days and conducts a fresh crawl of more than 3 million important Web pages each day. Google’s news crawl provides up-to-the-minute headlines for news queries, and a subset of its fresh news content is available here.

FAST and Fresh

How would you like a snapshot of the past year’s trends and a glimpse into the search engine future? To gather the information needed to take such a look, I’ve interviewed some of the major engines: AllTheWeb, AltaVista, Google, LookSmart, Lycos, and Yahoo I’ll be sharing what I’ve learned, beginning with AllTheWeb, a site owned by Fast Search & Transfer (FAST).

By offering AllTheWeb as a public search engine, FAST aims to provide users with comprehensive results while using the engine as a demonstration piece for its original equipment manufacturer (OEM) partners. The public search engine enables the company’s corporate partners to evaluate the technology before rolling it out on their own sites.

“FAST attacks the four pillars of search: relevancy, freshness, size, and speed,” said Rob Rubin, FAST general manager, Internet business unit. “This year, FAST introduced its linguistics tool kit to automatically detect phrases and provide accurate search results.” (Check this out by entering “Who is Little Richard?” or “What is the weather in New York?”)

Fast and Fresh, But Is It Profitable?

FAST’s business model is different than most search engines, Rubin explained. “The same search technology developed by FAST to power Internet portals on the Web (such as Lycos and T-Online) is used to provide ASP-based hosted search solutions for major corporations such as IBM,” he said. “We distribute our technology in shrink-wrapped packages and OEM for companies such as Reuters and TIBCO. In addition, we supply our technology to major e-commerce vendors, such as eBay and BroadVision.” This sounds more like a technology company than a search engine.

“Because AllTheWeb is a site where FAST delivers and, quite frankly, experiments with its technology on a significant number of sophisticated Web users, we have modest goals in its monetization,” said Rubin. “With AllTheWeb, we are looking to cover costs — not to drive the site itself to profits. That is for our OEM partners to do with their destination sites.”

But FAST did, in fact, make money. Its third-quarter revenue of $9.6 million, compared to the second quarter’s $8.7 million, was an increase of over 10 percent — not shabby considering last year’s meltdown. Rubin said he expects the same growth in the fourth quarter. When FAST released its third-quarter numbers, year-to-date revenue came in at $25.6 million, over five times the total revenue for the same period of the previous year.

In 2001, FAST trimmed its overhead and sharpened its focus toward revenue-producing product suites, such as FAST Web Search and FAST Data Search. By the end of 2001, the workforce headcount went from 230 to 170 and management resources became concentrated in Oslo, Norway, and Boston, due to some office closings in the U.S., Norway, and the U.K.

Creative Monetizing

FAST has two business units: the Internet business unit and the enterprise business unit. The Internet unit, which includes AllTheWeb, competes with the likes of Google, Northern Light, Inktomi, and AltaVista. The enterprise unit, which powers Internet portals and provides OEM solutions to major corporations, competes with companies such as Google, Inktomi, Autonomy, and Verity. FAST’s major income comes from the enterprise unit, which accounted for 60 percent of its revenue in the third quarter.

But efforts are also underway to monetize the Internet unit, namely AllTheWeb’s paid-inclusion PartnerSite service. That service guarantees insertion in the company’s databases (without influencing positioning or ranking) for a per-URL fee.

PartnerSite includes a 24-hour refresh and is not recommended for small sites. It can be a good thing for those who take the time to monitor and rewrite each page every 24 hours to achieve high rankings, but most in-house search engine optimization (SEO) operations lack the staffing for such redundant monitoring and modification.

With the PartnerSite paid-inclusion service available through FAST’s partner Lycos, customers are able to guarantee fresh inclusion in AllTheWeb’s index, have their site searched by the same search technology that powers major corporations such as IBM, and get distribution through all of the FAST customers.

Indexing the Deep Web

What has FAST done to index more of the “deep Web”? It completely rewrote and relaunched its crawler technology in November 2001. FAST can index many document types and can handle all kinds of dynamic content. For example, documents in PDF format are indexed at Scirus.

FAST’s PartnerSite inclusion program allows companies that want different content types indexed to be included on the Web. For example, through an inclusion program, IBM has the option to decide which of the documents on its public Web site it would like available via the index. (Actually, many of these different document types are currently available via an FTP search on AllTheWeb.)

To support news crawling, FAST has created a real-time cluster located in its data center. This cluster is based on FAST’s DataSearch product, which is the same product being used to power Reuters. FAST uses this technology to continuously crawl 3,000 premier news sites, keep the index around for five days, and continually refresh it every two hours. As new documents are crawled, the index is updated in subsecond latencies.

Tips for Good Rankings

Want high rankings? FAST recommends that technicians stick with the basics and work on leading-paragraph content. Use great design, title text, clear descriptions, and linkage from good reference points inside the network. SEO technicians should focus on improving content on the site. FAST will first check for Open Directory Project (ODP) descriptions; if none, it will look at meta descriptions; if none, it will examine the first 250 characters. Linguistic analysis of keyword frequency often reveals funny patterns — too many adjectives or an overuse of keywords. The document will then be flagged as spam.

FAST doesn’t like cloaking or spam and will fight vigorously to eliminate both — including black listing offending Web sites. If freshness and comprehensiveness are important to you, FAST suggests using its PartnerSite inclusion service. FAST reports that it constantly adjusts its relevancy algorithms to bring the freshest, most relevant content to its searchers.

Has Search Behavior Changed?

We’ve all heard that 86 percent of users find Web sites through search engines (GVU WWW User Surveys). Other studies report 42 percent or lower. Which to believe? Rubin recently read some Jupiter reports indicating that sophisticated users turn to search engines and destination sites, making search activities an integral part of the user experience. According to Jupiter’s Rob Leathern (August 2001), users go to search engines more when they are experienced. Approximately 30 percent of e-commerce purchases start at a general-purpose search engine. “FAST knows from its log investigations that people are launching their searches and navigating more using the Web engine,” said Rubin.

Engines of the Future

In the future, Rubin believes we’ll see more navigation features like “FAST Topics,” which are similar to the ODP topic listings seen at the top of the page and help searchers further refine their searches within specific categories. There is a demand for access to more Web content, and many companies want to make such content available. Rubin believes we’ll soon see more specialized catalogs like the ones FAST created for science at Scirus and for soccer information at Megasoccer.com. The focus for search companies will be on better intelligence to guide users to the results they are looking for. That will include the use of categorization and phrase matching with larger phrase dictionaries. Rubin predicts a period of great innovation and says that FAST is posed to lead the way.

Meta Search Engines

Last month, I discussed surviving in the search world and strategies used by second-tier search engines. Since you can get excellent exposure in meta-search engines, I’d like to provide more information on these tools.

We touched on how meta-search works and its pros and cons, but do you know the real strength of a meta-search? It’s powerful because one search can highlight strengths of a number of top search portals, such as Google, AltaVista, LookSmart, and Yahoo

You cannot be listed in meta-search engines. They don’t maintain a database. If you follow my prescription in my article “Getting Listed in Major Search Engines and Directories,” you’ll find yourself appearing in meta-search queries to great advantage.

Why is this important? Until recently, traditional search engines only indexed a fraction of the Web. Meta-search engines covered much more. That’s changing now that Google, AllTheWeb, and others are indexing deeper into Web content. You still search more of the Web with meta-search, because you are querying multiple databases. Nielsen//NetRatings said that 23.4 million unique users visited meta-search engines in October.

Who Queries What?

Below are the source lists for Ixquick, MetaCrawler, Dogpile, and Mamma.com, four of the major meta-search engines.

Ixquick

Ixquick is powerful and fast, with excellent search results. It provides a different source list for each query, depending on how various search engines ranked the Web sites containing the term queried. For instance, a recent search for “digital camera” resulted in the source list below, which includes Yahoo, LookSmart, AOL, and MSN, among others:

• AllTheWeb: robot search engine
• AltaVista: robot search engine
• AOL: indexing engine
• Excite: robot search engine
• FindWhat.com: pay-for-placement search engine
• LookSmart: Internet directory
• Lycos: robot search engine
• MSN: indexing engine
• Open Directory: Internet directory
• Overture: pay-for-placement search engine
• Sprinks: pay-for-placement engine
• Yahoo: Internet directory

Ixquick determines relevancy according to a star system, awarding each found Web site one star for every search engine that placed it in its top 10 for the query. It’s reasoned that relevancy is indicated when multiple engines agree a site is tops, all reaching that same conclusion with different algorithms.

MetaCrawler

MetaCrawler is an old stand-by. Its searches include Google, AltaVista, and LookSmart, in addition to the other databases listed below:

• AltaVista: robot search engine
• Direct Hit: question-asking directory (Ask Jeeves)
• Excite: robot search engine
• FindWhat: pay-for-placement search engine
• Google: robot search engine
• Kanoodle.com: pay-for-placement engine
• LookSmart: Internet directory
• Open Directory: Internet directory
• Overture: pay-for-placement search engine
• Sprinks: pay-for-placement engine
• WebCrawler: crawler-based search engine

MetaCrawler sorts and displays results by relevancy and eliminates duplications.

Dogpile

Dogpile has also been around for a while. Its searches include AltaVista, LookSmart, and Yahoo, as well as the other databases listed below:

• ah-ha: pay-for-placement search engine
• AltaVista: robot search engine
• Direct Hit: question-asking directory (Ask Jeeves)
• Dogpile Web Catalog: robot search engine
• ePilot.com: pay-for-placement search engine
• FindWhat: pay-for-placement search engine
• Kanoodle.com: pay-for-placement search engine
• LookSmart: Internet directory
• Open Directory: Internet directory
• Overture: pay-for-placement search engine
• RealNames: editorial/pay-for-placement system
• SearchHippo.com: database allowing users to add free keyword listings
• Sprinks: pay-for-placement engine
• Yahoo: Internet directory

Results are all-inclusive, sorted by search engine.

Mamma

Mamma’s searches include LookSmart, Lycos, MSN, and numerous file types, including MP3, image, and video files. It uses the following databases:

• 7Search.com: pay-for-placement search engine
• About.com
• Ah-ha: pay-for-placement search engine
• BIGwhat.com: pay-for-placement search engine
• Direct Hit: question-asking directory (Ask Jeeves)
• Image:Include: Image search engine
• Kanoodle.com: pay-for-placement engine
• LookSmart: Internet directory
• Lycos: robot search engine
• Mamma Collections: proprietary database
• MSN: indexing engine
• Overture.com: pay-for-placement search engine

Mamma sorts and displays results from the databases listed above, ranked by relevancy. Relevancy is determined first by rating the source engine (e.g., Yahoo rated higher than Lycos), then by assigning weights and comparing and ranking results from all the sources.

Because meta-search engines do not have their own databases, results are only as good as the quality of the sources the engines query. As they improve their capabilities and become better known, the number of people using them will continue to grow. Ensuring that your Web site is found not only by search engines but also by meta-search engines can help you tap into an audience of millions.

SEO and the Web Site Design Process

Does it make sense for Web site designers to consult with a search engine optimization (SEO) firm during the design and maintenance of client Web sites? That depends on various factors, including the size and level of sophistication of the site and client goals and objectives.

In some cases, a client can be stuck with unexpected costs if SEO coordination is not conducted early. For instance, even though a site is nicely put together, is functional, and features many products, search engine visibility may be hampered because the site uses frames or is primarily done with Flash using a limited amount of text. Search engines like to see as much text as possible — text that includes your strategic keywords.

Search engines work with text. They read the content on your pages and in your title and meta tags, recording this information in their database. Without text, the search engine has nothing to work with. Furthermore, text on pages that the search engines can’t get to doesn’t do any good. The search engines must be able to follow links from your home page to the other pages on your site that contain good content.

Someone once said, “A Web site is like a sand castle, not a bronze statue.” In other words, Web sites are works in process and you should be prepared to make or deal with changes on a regular basis. In almost all cases, deciding to integrate good search engine practices before, during, or after the design phase is irrelevant. What is relevant, however, is that site designers and their clients develop a plan for integrating search engine marketing and SEO strategies at some point before, during, or after the site is completed.

What are some of the search-engine-friendly basics to consider while designing a Web site?

• Give the engines plenty of keyword-rich text to work with. Text near the top of the page (when you look at the HTML source code) is more important, so put the text as high possible. Also remember to use your keywords in the text as much as possible, integrated into content that reads well. It’s important to place keywords at the beginning of paragraphs and headings. Write your text first for people and second for search engines. Bolding some of the keywords can also help.
• Make sure the search engines have plenty of basic links to follow. The search engines will index the text on your home page and then attempt to follow the links on your home page to other pages in your site. This process is called “spidering” or “crawling.”Most search engines cannot follow dynamic links, that is, links with a question mark in them. You must provide basic links, such as a href=”aboutus.xxx” or image maps. Dynamic links and links generated by JavaScript are not followed by most search engines.

  The text in the links is very important, too, and should include a keyword related to the page to which it is linking. A site map is a good way to provide links for the search engines.

• Provide a site map. A site map can make it easier for a search engine spider to find all your pages. It is basically a categorized list of all the pages on your site. There must be one or more links to the site map from the home page:
  • Make sure you have a text link to the site map at the bottom of the page. The text should say “Site Map,” and the file should be called “sitemap.htm.”
  • The link text should have strategic keywords where appropriate.
  • You should make a static list of links to all your pages, with a link to this page from the home page or the site map.
• Provide hypertext links. Ideally, links should have a keyword in the link text or in an .alt description of the image or area tag.
• Provide meta tags. Always provide a title, meta keywords, a meta description, and a descriptive comment tag in the head section of all indexable pages. The titles, keyword, and so forth should be unique if possible and apply to the particular page.Items in the header section should be in this order, using HTML:
  1. The title — 5-10 words with the company name and keywords
  2. The description — the description in number 3 or something longer
  3. The meta name description contents — 20-40 words with keywords
  4. The meta name keywords contents — a keyword list up to 900 characters
  5. Other optional meta tags
  6. JavaScript src tags

At the end of the day, it doesn’t really matter exactly when an SEO specialist gets involved in the Web site design process, although consulting with an SEO firm early can help you avoid critical mistakes. At any rate, it is crucial for all Web designers and their clients to budget for SEO modifications before getting too carried away with marketing and advertising. Search engine traffic will provide some of the most targeted visitors you can hope to attract, so it simply makes good sense to optimize for search engines before blowing the entire marketing/advertising budget on paid advertising and offline media.

Surviving in Today’s Search World

Search engines, like other dot-coms, have encountered choppy seas since ad revenues went south. Many are struggling for survival. All are looking for new ways to monetize.

Yahoo is cutting operating expenses with layoffs and paid subscriptions and services. Most astonishing was a deal with Overture to display paid listings, along with plans to launch a paid placement program in the spring. Fans may not like it, but the firm waited too long to change its business strategy and lost its dominant position. Nielsen//NetRatings shows Yahoo trailing AOL in the top 25 Web properties for November.

Search engines surviving the storm were not unscathed. Still, many endured. Those struggling include LookSmart, AltaVista, Infoseek, and Excite (ExciteAtHome is in bankruptcy). Those running a tight ship include Google, Overture, FAST, AOL, MSN, and Mamma.com (to name a few).

Navigating in Troubled Waters

What does it take to remain profitable in hard times? Lesser-known, middle-tier search engines such as Mamma.com have remained profitable by sticking to the knitting and capitalizing on core assets. When asked what makes Mamma.com successful, Marketing Director Davide Molino replied, “Middle-tier engines have survived by maintaining small staffs with reasonable salaries, keeping their sites simple, listening to employees who are in the trenches every day, running with new ideas, and, most of all, being honest with users.”

Mamma.com went live in 1996 and grew slowly through word of mouth. It currently reaches 8 million unique users per month through its network of sites. At the height of the dot-com frenzy in 1999, it employed a banner sales force and business development team, and advertised heavily with offline billboards and TV spots. This search engine recognized its niche and didn’t become a me-too portal.

Recognizing a trend, Mamma.com started selling search result rankings with its “Premium Search Results” program, shuffling paid listings in with meta-search results. It later labeled them “Mamma Recommends,” when search engines were criticized for not fully disclosing advertising. They were subsequently renamed “Mamma Classifieds,” to clarify that they are paid listings. Further diversification includes a proprietary banner delivery system to replace DoubleClick’s DART (Advertising Delivery System, a.k.a. ADS) and an automated registration system slated for launch in the spring.

Mamma.com’s strategy of capitalizing on core assets seems to be paying off. It will provide search functionality for InternetNews.com and EarthWeb as internet.com (ClickZ’s parent) provides a “Search the Web” option on those sites. An internet.com user selecting “Search the Web” will be linked to a Mamma.com search result page with the option to continue searching or return to internet.com.

Meta-Search Explained

In case you don’t know, a meta-search engine queries multiple sites to provide results, and does not maintain its own database. When you type in a query, it simultaneously searches a series of other sites and compiles and displays those results. Because they don’t maintain a database, meta-search engines do not accept URL submissions.

In using a meta-search engine, you get a snapshot of the top results from a variety of search engines. Proponents say these tools return fewer results with a greater degree of relevance. Critics say the searches are quick and dirty, because they don’t query all the best sources (such as Google and Northern Light) and don’t allow refined searches. Mamma.com claims to return results 96 percent of the time, displaying data from five different sources (on average). Besides Mamma.com, other meta-search engines include Copernic, Dogpile, Ixquick, MetaCrawler, ProFusion, and Viv&iacutesimo, to name a few.

Monetizing the Search

Most search engines and directories have adopted some means of making money from presenting search results. Paid links were considered a threat to editorial integrity, so the first monetizing effort by major search portals was the paid submission/inclusion program started by Yahoo These programs require payment to expedite entry of Web sites/pages into a search database.

Paying for timely consideration allows the editorial product to remained untainted while search engines gain revenues. Examples of paid inclusion programs are Yahoo’s Express, LookSmart’s Express Submit, Inktomi’s Search Submit or Index Connect, AltaVista’s Trusted Feed, and FAST’s PartnerSite. The engines that don’t use paid inclusion are Google, Open Directory Project (ODP), and Northern Light, although Google provides AdWords as paid placement.

When it launched, no one would have predicted Overture’s phenomenal success. Despite the fact that a year ago paid listings were widely condemned, they’ve gained a foothold and now seem to be accepted as long as links are clearly labeled as advertising.

Staying Competitive

Based on information from the recent Search Engine Strategies 2001 conference, the current industry focus is on freshness and indexing more data of various file types from the deep Web to entice users.

Many search engines are improving database freshness, selectively updating pages that change frequently by crawling them more often. FAST completely refreshes its database every 9 to 12 days, claiming to be “the Internet’s freshest and most up-to-date search engine.” Inktomi presently re-indexes its “Best of the Web” every 9 days (about 20 percent of its database). Google’s spider crawls the entire Web (about a billion pages) every 28 days, crawling news sites and such more often.

Search engines are eager to index beyond HTML and text page results. They’re seeking better ways to integrate relevant information from Web pages, images, video files, MP3 files, Usenet messages, maps, news, and FTP files all in one search, showing multiple results on the same page. FAST started doing this in August; others, such as Google and Lycos, are not far behind.

Survival Guide

You don’t have to be a top banana or household name to succeed in the search engine game. It helps to have a good business plan, to identify your niche and industry trends, to be decisive, and to move quickly.

Search Engine Optimization for Everyone

Section 508 of the Americans with Disabilities Act (ADA) requires federal agencies to ensure that all electronic and information technology used is accessible for people with disabilities. This includes Web sites. Though the public sector has no such requirement, a site designed for accessibility will broaden your audience and improve your market share, because accessibility is not only beneficial to disabled users but also to the growing number of users with electronic devices such as personal digital assistants (PDAs), cell phones, and Wireless Application Protocol (WAP) devices.

The more accessible your Web site, the wider your potential audience. Think about it: Can you afford to lose the 28.2 million potential prospects who are color blind (predominantly men) because you haven’t considered the effect of color on their ability to use your site? Can you afford to inconvenience people who access the Web from different browsers, diverse environments, various devices or platforms, or low connection speeds? An accessible site is user friendly because of emphasis on content, structure, and ease of navigation — a big plus for users. Few things annoy Internet users more than not being able to accomplish their objectives effortlessly when visiting Web sites — be it the ability to find information or the ease of processing that information. Reaching a wider audience and accommodating customer needs are good reasons to consider accessibility.

Enhance Accessibility and Improve Usability

Search engine optimization (SEO) of your site for search engines enhances accessibility and improves usability, but benefits go further than that. It’s smart to ensure that your site is viewable and navigable from a wide variety of Web clients and for people accessing the Web from a variety of different environments. Here’s why:

• One out of every 12 men are color blind, thus your choice of colors affects their ability to use your site.
• Baby boomers surf the Web in large numbers. This demographic is the fastest growing group of new Internet users. These people find it hard to read the small fixed fonts frequently used by designers. They also are users of cell phones and PDAs.
• Not everyone has a mouse and keyboard — or the ability to use these tools. Increasingly, more people are using speech browsers while accessing the Internet in their cars. Others have difficulty using a mouse due to carpal tunnel syndrome or because of visual impairment.
• Not everyone can view graphics. Beyond the needs of visually impaired users, many people access the Web with slow connections, disabling graphics and scripting capabilities. Some may use a text-only screen or a small PDA screen. Some people still use older browsers and computers.
• Environments can influence accessibility. Some people work where lighting makes it difficult to read the screen. Others may be in situations where eyes, ears, or hands are occupied, interfering with certain tasks (e.g., people driving or working in a loud environment).
• The popularity of mobile devices is an accessibility challenge as more and more people use wireless devices to access the Web.

Add it up. Inaccessibility takes a toll in lost opportunities for prospects and sales. It’s important to plan ahead, taking the time to identify your site goals, intended audience, content, design, and SEO services. This will save money in the long run and makes the updating and maintenance of your site easier.

If your site is already up, have it audited (Bobby is a free service to help identify and repair accessibility barriers), then prioritize improvements by level of impact to meet the basic accessibility standards.

SEO Content Optimization and Accessibility

Content optimization and accessibility are complementary processes. When optimizing a site for search engines, a good SEO provider will analyze the site’s code, structure, and page layout, looking for any elements that might prevent a robot from indexing the site. Content optimization resulting in search engine friendliness also helps accessibility.

For example, a Flash site can benefit from creating a static mini-site of a few pages with good text, providing links to the Flash site on each page. The mini-site is accessed as a static page from the main site. It may not be as visually engaging, but the text is readable without Flash. Compensating for “bells and whistles” makes the site crawlable by search engine robots and gives users who can’t view Flash the ability to access a low-bandwidth alternative.

Accessibility Guidelines

The W3 Web Content Accessibility Guidelines’ Table of Contents tells you how to make your site accessible. Each issue is described in detail, and examples are provided. The guideline headings are:

1. Provide equivalent alternatives for auditory and visual content.
2. Do not rely on color alone.
3. Use proper markup and style sheets and do so properly.
4. Clarify natural language usage.
5. Create tables that transform gracefully.
6. Ensure that pages featuring new technologies transform gracefully.
7. Ensure user control of time-sensitive content changes.
8. Ensure direct accessibility of embedded user interfaces.
9. Design for device-independence.
10. Use interim solutions.
11. Use W3C technologies and guidelines.
12. Provide context and orientation information.
13. Provide clear navigation mechanisms.
14. Ensure that documents are clear and simple.

It’s advantageous to make your site accessible to broaden your audience. Enhancing accessibility with SEO services makes sense because good positioning helps improve market share and branding.

More Accessibility Resources

• IBM Accessibility Center — Checklist and references for Web accessibility.
• The Alliance for Technology Access — A network community providing information and support services to people with disabilities.
• WebABLE — Provider of Web accessibility technology, consulting, and training. The library includes press releases, white papers, articles, and references on accessibility and adaptive technology.
• AWARE Center (Accessible Web Authoring Resources and Education) — HTML Writers Guild central resource for educating Web authors on accessibility.

Selecting a Reputable SEO Provider

I often get questions about selecting search engine optimization (SEO) providers. Since I’m speaking on “Selecting an SEO Provider” next week at Search Engine Strategies 2001 in Dallas, I thought I’d cover this topic and invite readers to pose questions or comments for the panel.

There are many issues to consider — more than I can cover in this space: definition of services; pricing; vendor experience and proficiency; customer service; and contracts and reporting. MarketingSherpa’s “Buyers’ Guide to Search Engine Optimization & Positioning Firms” covers this in detail and reviews reputable SEO providers. ClickZ’s sister site Search Engine Watch also contains a lot of valuable information. Below are my own suggestions for gathering information on potential providers.

Definition of SEO Services

Business sites considering professional SEO, positioning, and manual submissions can choose from services in four categories: site analysis and optimization, manual submissions, offsite development, and monthly reporting.

1. Site Analysis and Optimization

• Category and keyword research. This requires development of strategic keyword phrases from client suggestions and further research and analysis. The number of terms is proportional to the campaign cost. Reevaluation of keyword phrases should be included.
• Home page optimization. This requires specific HTML optimization of your home page and subpages (title, meta-tags, alt tags, keyword usage) and general site suggestions (navigation, subpage guidelines, site map usage).
• Additional content suggestions. These ensure that your strategic keywords are reflected throughout well-written site content — including a review of content quality and placement — suggesting improvements.
• Directory compliance issues. Directories such as Yahoo, LookSmart, and Open Directory Project (ODP) have specific guidelines for good placement. Directory compliance issues ensure site compliance, suggesting necessary changes.
• Dynamic site suggestions. Many search engine spiders can’t crawl dynamic sites. This requires home-page optimization, site modifications (creating a few crawalable dynamic pages), additional content creation, and specific subpage submissions to certain directories.
• Frame site suggestions. Subpages within framed sites are sometimes incompatible with search engines. This may require site modifications to ensure subpage indexing and that subpage listings will call up your framed site correctly.

2. Manual Submissions

• Submission of home page and subpages to top engines and ODP. This requires working with ODP editors to acquire the best possible category listing, as ODP provides search results for Netscape, AOL, and many others. It includes submission and monitoring of your home page.
• Paid directory submissions. As a critical element of any SEO campaign, directory submissions require very special care once a year. These submissions necessitate decisions on which inclusion submissions are necessary for your positioning campaign.
• Yahoo Business Express. A Yahoo Business Directory listing is mandatory. It requires working with Yahoo editors to acquire your best possible category and submission in compliance with guidelines (keyword density and placement in the title, description, and URL).
• LookSmart Express Submit. A LookSmart listing is important because of daily high-volume searches (60 million per day through 5 of the top 10 portals, 5 of the top 10 Internet service providers, and hundreds of media partner sites worldwide). It requires working with LookSmart editors to acquire your best category and precise submission.

3. Offsite Development

• Keyword optimized pages. High rankings can be obtained by delivering a specific Web page to a particular engine seeking that information. This service requires creating, submitting, monitoring, reporting, researching, and resubmitting each keyword-phrase-optimized page for each keyword phrase to each individual robot engine.
• Robust link popularity. Professional technicians can provide specific and relevant content links created from content-rich Web sites linking directly to your site. This service requires analyzing your site to determine and match your content with relevant content from cataloged content pages, online publications, and topic-focused articles.

4. Monthly Reporting

• Search-based position reports. This service requires reporting positions for each URL and keyword phrase once a month. You’ll see exactly where your home page, subpages, and optimized keyword pages are located. Reports list the page number and location (numbered 1 through 50) of each keyword phrase search.
• Log file analysis. This service requires evaluation of your server logs to identify your SEO results, which can then be used by your staff as relevant statistics.

SEO Pricing

All the above services can be considered when pricing an SEO campaign, but remember: One size does not fit all. Some industries require more SEO resources than others. Ask specific questions about pricing of the above services. You should get an itemized proposal specifying one-time versus monthly costs. Each recommended service should be clearly described in a technical support document explaining the need for every item in your campaign.

SEO and positioning require a long-term commitment and realistic understanding of the technical and human resources required. Technicians do not work for minimum wage, and SEO systems require significant investment and maintenance. Most campaigns average $12,000 to $60,000 a year, whether in-house or outsourced.

Vendor Experience

Core competency is important when selecting an SEO provider. Ask the following questions to determine the experience and proficiency of the vendors you are considering:

• How many years of SEO experience do you have?
• Do you provide other Internet marketing services, or do you specialize in SEO?
• What is your company’s proficiency in SEO? Do you specialize in specific industries?
• What is the cost breakdown for these services (setup/maintenance, paid inclusion, etc.)?
• Will the work you do impact my existing site?
• What clients have you served, and what results were achieved?
• Can we contact these clients?
• How long does it take to achieve results?

Customer Service

This is an important but often overlooked issue. You should be able to discuss your SEO plan directly with a qualified search engine technician or an account manager who understands every aspect of your campaign. You need to know what support services are provided and what will cost extra. Consider asking these questions:

• Will I have one-to-one, direct contact with an SEO account manager?
• What customer support services are provided?
• How do you provide support (phone/email), and is there a limit (time/number of emails)?

Contracts and Reporting

Find out what the minimum contract includes, the cost of consulting, whether any guarantees are offered, and the hosting of domains used. SEO results are not usually guaranteed because they change continuously. That’s why it’s important to find out if maintenance and reporting are provided. Ask the following questions:

• Do you require a minimum contract and if so, how much?
• Do you charge extra for consulting or reoptimization?
• What maintenance do you provide, and what is the monthly cost?
• What reports do you provide and how often?

Making the Right Decision

Gathering the information above and comparing vendors should help you make the right decision. So would attending Search Engine Strategies 2001 next week in Dallas, if you’re able. I hope to meet many of you there!

Understanding the Power of Search Engine Marketing

CEOs and marketing directors focus on strategic, not tactical, goals. They are often unaware of their company’s search engine rankings. Those who approve marketing budgets need to know the power of search engine marketing. It’s one of the most cost-effective marketing tools available, yet most companies spend little on search engine optimization (SEO).

Users find Web sites via search engines — a fact that’s been documented time and again. Searching is second only to email as the most popular online activity. How do consumers find products online? According to Jupiter Media Metrix (September 2001):

• 28 percent type a product name into a search engine.
• 23 percent go straight to the URL.
• 9 percent type a brand name into a search engine.
• 5 percent type a company name into a search engine.
• 5 percent go to a search engine shopping channel.

A good chunk of business can come from search engines. It pays to know how they work and how they can improve your bottom line.

Search Engines Versus Directories

We all use Yahoo and Google, but does everyone know the difference between a directory and a search engine?

Directories, such as Yahoo, use human editors to evaluate sites for acceptance based on content relevance and uniqueness. Directories are organized by categories, then broken down into subcategories. Once a listing is accepted, the relevancy algorithm determines the ranking. The way technicians use keywords to write titles and descriptions is critical for top rankings. It’s also important to get your site listed in the right category. This process can be complex, as I described recently.

The major directories are Yahoo, Open Directory, LookSmart, and Ask Jeeves. Open Directory feeds results to AOL, Netscape, Lycos, Google, HotBot, and many more. LookSmart feeds results to MSN, Excite, AltaVista, iWon, Netscape, and others.

Search engines, such as Google, use different criteria for listing Web sites in their databases because they “crawl” or “spider” the Web. They index pages with software designed to return relevant results. If your site is search engine friendly it will rank higher, meaning it will show up higher on the list of search results. It matters which engines and directories you submit to first. Though all search engine algorithms differ, there are similarities. The algorithms are frequently changed to prevent unethical practices.

The major search engines are: Google, Inktomi, AllTheWeb (FAST), AltaVista, and Excite. Google feeds secondary results to Yahoo and Netscape. Inktomi results go to AOL, HotBot, LookSmart, and MSN. FAST feeds Lycos. AltaVista gets secondary results from LookSmart, as does Excite.

Why Invest in SEO?

Search engine traffic is very targeted. Potential buyers who find your site via a search engine are actively looking for your products and services. They are literally asking for your marketing message. Case studies show that companies can land big clients through their Web sites alone without having to make house calls. Software company Welocalize reported 90 percent of site traffic is driven by search engines, with traffic patterns ranging from 5,000 to 10,000 user sessions per month.

Database software firm Syncsort generates over 75 percent of its leads from the Web via search engine listings, according to iProspect, a consultant for the company.

Automotive consumer resource site Edmunds.com, another iProspect client, logged 1.2 million visitors from search engines and directories last June, attracting over 7 million visitors from search engines over the past year.

SEO is cost-effective compared to other advertising. A key-phrase banner ad campaign can run from $2,500 to $35,000 per month, depending on the number of key phrases. A professional SEO campaign with an equivalent number of key phrases will cost anywhere from $1,000 to $10,000 per month.

Search engine text links can be a powerful branding tool. A study by NPD Group reported that search engines outperformed banner ads three to one. More than twice as many respondents recognized companies listed in the top three search positions than companies featured in banners ads.

Search engine traffic can produce high conversion rates. Case studies based on client-server file logs show that search engine traffic outperforms banner ads and email marketing campaigns by producing more purchases, form or catalog requests, newsletter subscriptions, and software downloads.

Should You Outsource or Optimize In-House?

Though many firms perform SEO in-house, research shows this may not be the best option:

• An iProspect study found that 97 percent of Fortune 100 companies have chosen to use some sort of technology that makes it difficult for search engines to find their sites.
• CyberAtlas reports that most marketers spend less than 1 percent of their budget on SEO, even though optimization can increase site traffic 75 percent or more.

The Outsource Advantage

An expert can get better rankings faster. SEO experts focus on submission and maintenance full time, offering a range of choices in methodologies and pricing that result in the right strategy for your Web site and industry. SEO is a complex process that requires expertise to maintain long-term results.

SEO has a steep learning curve because every search engine has different ranking criteria, and algorithm changes are ongoing. When you outsource, you avoid resubmission errors that can get you blacklisted on the engines.

Search engine positioning changes daily. The cost of recurring research and methodology development to maintain listings can turn the tables on in-house profitability. A reputable SEO firm is better prepared and has the flexibility to deal with changes.

Outsourcing is cost-effective when compared to in-house efforts because SEO firms can take advantage of economies of scale after the initial investment of time and knowledge, methodology, development, and technology. In-house investment by individual Web sites can be substantial and lack cost maximization.

A reputable SEO firm will provide up-to-the-minute options, recommendations, and technical advice for Web site changes to improve rankings. Typically, in-house efforts are behind the curve because most instructional material published on SEO tactics is obsolete shortly after it’s published.

Selecting a Provider

When outsourcing, first define the services to be included in your SEO campaign, such as:

• Complete keyword-phrase research
• Manual submission to major search engines and directories
• Paid inclusion to major search engines and directories
• Monthly reporting
• Meta-tag optimization
• Page title and headings optimization
• Reciprocal linking/link popularity
• Body text editing and copywriting (by page)
• Consulting on creation of new pages or alternative text within your site
• Consulting on creation of information, gateway, bridge, and off-site pages

Boost Your Bottom Line

SEO is both an art and a science. A reputable SEO provider can increase your site visibility (number of page-one listings) dramatically. Optimization tactics are complex and time consuming; that’s why savvy companies outsource these services. If you’re wondering who to hire (an equally complicated task), check out “Buyers’ Guide to Search Engine Optimization & Positioning Services” for profiles of reputable SEO providers.

New Yahoo! Formula Requires Different Optimizing Strategies

People have been complaining for years about search engine relevancy. Finally, the industry is responding. Users will love these changes because they’ll find more relevant content. The trend toward relevancy was pioneered by Google, unofficially dubbed “the best search engine on the Web.” A spate of new search technology that permits indexing of deep Web content also provides users with more relevant results, a trend that continues to gain momentum.

How does this affect search engine optimization (SEO)? Yahoo changed three basic factors in its algorithm formula that affect positioning of categories, Web sites (directory listings), and Web pages (Google listings): alphanumeric importance; keyword use in title, description, and URL; and category weight.

Alpha No Longer Rules

Yahoo has finally eliminated emphasis on alphanumeric order (like the yellow pages). Top-listed sites formerly had names such as AAA Diabetes Support Information, Ace Hardware, or 1st Choice Marketing Promotions.

Yahoo’s directory no longer sorts arbitrarily by alphanumeric standard. The new default search will sort data more like a search engine, determining relevancy based on a specific set of algorithmic factors.

Keywords Are More Important Than Ever

The emphasis is now on keyword density and placement in the title, description, and URL. Yahoo provides default search results weighted more heavily toward these factors than alphanumeric and category factors.

Density. Consideration is given to the ratio of keywords versus nonkeywords appearing in your title, description, and URL. For example, consider the hypothetical title, description, and URL below:

• HTML Title: Diabetes Support Group
• Description: provides information on Diabetes etiology, treatment, and support resources
• URL: www.AAADiabetesSupportInformation.net

For the search phrase “Diabetes information,” your title keyword density would be 33 percent (one out of three words match). Your description keyword density would be 22 percent (two out of nine words). Your URL keyword density would be 50 percent (two out of four words). If the URL were simply www.Diabetes-Information.net (eliminating AAA), you would improve relevancy considerably to 100 percent keyword density. Note the hyphen (more on that later).

Tweaking the title and description can improve relevancy, but you can’t tweak your URL (without changing your domain name). Don’t get carried away by repeating keywords needlessly. Yahoo editors are known to edit copy, especially if it has redundant keywords. It’s more important than ever to write your page’s HTML title and description carefully to create high-relevancy ratings while not inviting a change by Yahoo editors. That’s why I suggest a brief description rather than a lengthy one, to avoid outside editing that risks the removal of keywords.

Placement. The location of your keywords in your title, description, and URL is also important. In the past, you were advised to use a variety of different keyword combinations in your title and description to enhance relevancy for a number of different searches.

Now, it matters where you put these keywords. The closer your keyword phrase is to the beginning of your title, description, or URL, the better. Having your keywords in your URL is now very important, and it’s best to separate them with hyphens in order to score well for all of them.

When searching for the keyword phrase “Diabetes information” in the URL www.AAA-Diabetes-Support-Information.net, both “Diabetes” and “information” will count toward relevancy if separated by hyphens. But if your URL is www.AAADiabetesSupportInformation.net, you won’t score for relevancy on either one of those keywords because the first word is AAA, and the keywords are not separated by hyphens. Yahoo assigns importance to the first keyword, and only to a subsequent keyword if it is separated by a hyphen.

Use a keyword as the first word in your title, description, and URL if possible, because that is now more valuable than any subsequent keywords. Keep keywords together. Yahoo’s new algorithm places more weight on the first word used in a searched phrase and on keywords grouped together rather than separated by nonkeywords. For instance, “current medical information on Diabetes” is less powerful than “current medical Diabetes information.”

I can’t overemphasize the importance of keyword research. You can use tools such as the GoTo Search Suggestion Tool, Wordtracker, or WordSpot to identify your most popular keyword phrases. This process is easy to oversimplify, resulting in mistakes that can penalize your rankings. For that reason, many marketers hire consultants for keyword research.

Keywords in your URL. You might consider purchasing some URLs in .net and .org extensions to include your keywords in your domain name if appropriate.

Main Categories Carry More Weight

You’ve all seen Yahoo’s 14 main categories and the subcategories within them. In the new algorithm formula, the category closest to the main category now carries more weight. For instance, a search for the phrase “Diabetes information” would score better if the site were listed as shown in the first example (hypothetical) than it would if listed as shown in the second example (actual):

• Business and Economy > Shopping and Services > Health > Diabetes Information
• Business and Economy > Shopping and Services > Health > Diseases and Conditions > Diabetes

If your keyword phrase appears in a category name, categories closer to the main category carry more weight.

Results Are Displayed Differently

There is a noticeable change in the way Yahoo displays results. You will note that the site now offers searchers a choice between “Category Matches” and “Web Site Matches.” In the past, results defaulted to the directory “Categories” when answering search queries. Now the results default to “Web Sites.” This change has implications for Web page design.

So how should you design your pages — optimized for search engine results or alphanumerically friendly for category results? The short answer is to design for search-engine friendliness. More people are familiar with simple search strategies than with navigating a directory through categories.

Implications for SEO

You need more expertise than ever to optimize and list your site on Yahoo in light of this new algorithm formula. Besides expert keyword research, you need pro copywriting to nail the title and description. And since keywords in the URL are important, you might need domain name consulting. Research suggests that marketers could benefit from outsourcing SEO services.

Several SEO studies have been released recently, including those from CyberAtlas and MarketingSherpa.

It was interesting to read in MarketingSherpa that “more than 90 percent of the Fortune 100’s sites are NOT optimized properly (although they have in-house ‘experts’) and most marketers spend less than half of one percent of their budgets on optimization even though it can bring them up to 75 percent of their site traffic!” CyberAtlas stated that most marketers (nearly 70 percent) were less than satisfied with the Web traffic they currently receive from search engines.