Keyword Research for Organic SEO

So you have decided to venture out into the world of SEO. The first thing you will need to do is determine the direction of your campaign in relation to the key phrases you are choosing to target. This article will focus on how to find keywords for your organic campaign, as the process is slightly different for PPC.

Many site owners know immediately what phrases they want. If you feel like you know what you want, before you start take a brief step back and assess if this really is the best phrase for your site. Yes, it just may very well be the perfect phrase, but if it isn’t, you could wind up spending a lot of time and money pursuing a ranking that either will never happen, or will provide very little value to your site.

There are a few key areas to look at when choosing a target phrase:

1. Relevance – Is this phrase even relevant to your site and its content?
2. Search Frequency – Are people even searching for this phrase?
3. Competition – How competitive is this field? Is it even a feasible target?


Where to start – Create a List of Phrases
So where do you even start with all this keyword research. Before looking up search frequencies and competition you need to create a list of relevant phrases. Open up an excel sheet and type out all relevant phrases that come to mind, do a little brainstorming as there are no wrong answers at this state.

After you have exhausted your thoughts, move over to your website. Open it up and navigate throughout recording any keyword phrase ideas that spring up checking your title tags and body content. Once this is done, do the same thing with your competition. Visit some sites that you know are in direct competition with you and go through them recording any relevant phrases you see.

By now you should have a long list of potential targets, a list that will grow further as you look into their search frequencies.

Find a Keyword Tool
The next step is to open up your favorite keyword research tool. There are many to choose from, two of the more popular being WordTracker and Keyword Discovery, although many still use the free, Overture tool. It is important to note that no keyword tools give you 100% accurate search figures. In most cases you will get numbers representing a sampling from various search engines. These numbers are best used in comparing one phrase to another to find out which is more popular, rather than determining specifically how much traffic to expect.

Check the Search Frequency
Once you’ve opened up a keyword tool, begin entering your keyword phrases and record their noted search frequency. Be sure to scroll through the results recording any additional phrases that are both relevant and have acceptable search frequencies. The exact number of searches required to make a phrase acceptable depends widely on industry, and even the search tool being used. A phrase with only 100 searches per month may be perfect for a secondary target, but in most cases may not be the best bet for a primary phrase.

Sorting Your List
You now should have a very exhaustive list of potential target phrases and their corresponding search frequencies. Sort this list in descending order based on the number of searches, so that the most popular phrase is at the very top. In many industries, the top few phrases may be completely impractical to target due to the competition, but we’ll determine that a bit later.

Check the Competition
The next step is to get a feel for h
ow competitive these phrases are. In the next column in your spreadsheet, place the number of results returned by Google for each individual phrase. The lower the number of competing pages, in most cases, the easier it may be to achieve rankings. (Note: this is not always the case, but it is an indicator).

At this point, you will have a long list sorted by search frequency, along with the number of competing pages. If you are fortunate, you will see one phrase immediately that jumps out – solid searches with low competition. This just may be the most ideal target phrase.

Does this phrase fit well with the theme of your site? If so, go to Google and take a closer look at the ranking websites. Does your site fit in with the general feel of these results? In some cases it may not, as your phrase could have different meanings (especially true if using acronyms). This phrase may represent a completely different part of the world if geographically targeted, or simply may be littered with mega competitors such as eBay, Amazon, WikiPedia, and others. If you can see your site fitting in with these results, it’s time to assess the general feasibility of this phrase.

Take a look at the number of back links, and indexed pages each site has. Do your numbers compare? If you find that the top 10 ranking sites all have back links well into the tens of thousands, and your site has a dozen or so, you may want to consider a different phrase. If the ranking sites are in the high tens, or low hundreds, and your site has a dozen links, then you have something to work with, if you are willing to work on increasing your link counts. The number of pages indexed is less important than links, but if you have a 6 page site and you are planning on competing with thousand page sites, your chances of success will be much lower.

The real key is to try to find a phrase that offers relevance, decent searches, and competition that is not way out of your league.

Pick a Phrase to Drive Qualified Traffic
For organic SEO it is usually best to focus on one primary phrase that best suits your site, while targeting more specific secondary phrases for relevant sections of your site. With organic SEO, how many phrases you should target is somewhat limited by the size of your site, the larger the site, the more phrases you will have the ability to work towards.

The phrase with the most searches is not always the best fit. This is largely true with the real estate market.

Because everyone has free access, I will use the Overture Keyword Selector Tool for an example. The phrase “real estate” saw 3,057,037 searches in January of 07. On the surface this phrase seems like a dream come true, but you have to consider the geographic issues.

If your office serves the Seattle area, is someone searching in Orlando likely to be a qualified visitor to your site? In most cases no. Targeting the phrase “Seattle real estate” with 12,441 searches, seems like a much better choice as it would deliver more qualified traffic. While this phrase is still quite competitive, it is not nearly as difficult as simply “real estate”. Take a look at the big picture and determine not only how likely it is that you may achieve rankings, but whether the traffic generated from such a ranking would actually have a positive impact on sales.




Conclusion
Doing some research to find the best target phrase is the groundwork for your SEO campaign. Without it you’ll be flying blind with no clear direction on goals. Take the time up front to do a little research and determine whether the dream phrase you have in mind is a worthwhile target or not. If it turns out that it’s not, its better to find out before you invest your time and money on an SEO campaign. Knowing the level of competition and search frequencies for a target phrase beforehand will help you make informed decisions and give you the best chances for success.

Source: http://www.isedb.com/

The robots.txt file and search engine optimization

On how to tell the search engine spiders and crawlers which directories and files to include, and which to avoid.

Search engines find your web pages and files by sending out robots (also called bots, spiders or crawlers) that follow the links found on your site, read the pages they find and store the content in the search engine databases.

Dan Crow of Google puts it this way: “Usually when the Googlebot finds a page, it reads all the links on that page and then fetches those pages and indexes them. This is the basic process by which Googlebot “crawls” the web.”

But you may have directories and files you would prefer the search engine robots not to index. You may, for instance, have different versions of the same text, and you would like to tell the search engines which is the authoritative one (see: How to avoid duplicate content in search engine promotion).

How do you stop the robots?

the robots.txt file

If you are serious about search engine optimization you should make use of the Robots Exclusion Standard adding a robots.txt file to the root of you domain.

By using the robots.txt file you can tell the search engines what directories and files they should spider and include in their search results, and what directories and files to avoid.

This file must be uploaded to the root accessible directory of your site, not to a sub directory. Hence Pandia’s robots.txt file is found at http://www.pandia.com/robots.txt.

Plain ASCII please!

robots.txt should be a plain ASCII text file.

Use a text editor or text HTML editor to write it, not word processors like Word.

Pandia’s robots.txt file gives a good example of an uncomplicated file of this type:

User-agent: *
Disallow: /ads/
Disallow: /banners/
Disallow: /cgi-local/
Disallow: /cgi-script/
Disallow: /graphics/

The first line tells the robots which robots are to follow the “commands” given below this line. In this case the commands are for all search engines.
The next lines tells the robots which Pandia directories to avoid (disallow).

Lets take a closer look at the syntax for disallowing directories and files.

Blocking an entire site

To block the entire site, you include a forward slash, like this.

Disallow: /

This is not a procedure we recommend! If you want to block search engine spiders from crawling your site, you should make it password protected. The search engines have been known not to respect the robots.txt files from time to time.

Blocking directories

To block a directory and all its files, put a slash in front of and after the directory name.

Disallow: /images/
Disallow: /private/photos/

Blocking single files

To stop the search engine(s) from including one file, write the file name after a slash, like this:

Disallow: /private_file.html

If the file is found in a subdirectory, use the following syntax:

Disallow: /private/conflict.html

Note that there are no trailing slashes in these instances.

Note also that the URLs are case sensitive. /letters/ToMum.html is not the same as /letters/tomum.html!

Identifying robots

The first line User-agent: * says that the the following lines are for all robots.

You may also make different rules for different robots, like this:

User-agent: Googlebot
Disallow: /graphics/

Most web sites do not need to identify the different robots or crawlers in this way.

These are the names of the most common “bots”:
Googlebot (for Google web search)
Slurp (for Yahoo! web search)
msnbot (for Live Search web search)
Teoma (for Ask web search)

Source : http://www.pandia.com/sew/489-robots-txt.html