Learn WordPress
Home » SEO

Search Engine Optimization Using robots.txt

Appears on the first page of search engines with appropriate keyword are the desires of each web owner, but it is not easy because a lot of things to do in order to optimize a web page to be like the search engines wants. This time I would like to share a bit of how to limit the search engines to index so carelessly made no specific restrictions that must be obeyed by the search engine spiders = crawler in its job to index a web page. Rules are written in a file called “robots.txt“.

robots.txt

Most websites or blogs have files and folders that are not useful to index by search engines such as image files, admin files or files that you consider confidential. You can limit the spider not to index the file by creating a robots.txt file. To create this file is very easy, you only need a notepad to write robots.txt script. Example code can be viewed like this:

User-agent: *
Disallow: /images/
Disallow: /feed/

The code is not allowe spider to access and indexing the contents of the directory of /images/ and all URLs that begin with /feed/.

You can copy and paste the code below to create a robots.txt file

User-agent: Googlebot

Disallow: /log/*/trackback

Disallow: /log/*/feed

Disallow: /log/*/comments

Disallow: /log/*?*

Disallow: /log/*?

Disallow: /log/search

User-agent: *

Disallow: /cgi-bin/

Disallow: /log/wp-admin/

Disallow: /log/wp-includes/

Disallow: /log/wp-content/plugins/

Disallow: /log/wp-content/themes/

Disallow: /log/trackback

Disallow: /log/comments

Disallow: /log/feed

Or you can create robots.txt by using this free online tool service.

Next, placing robots.txt file in the root directory of your website.

Placement location of the robot.txt should be in the main directory = root directory for the search engines while your coming our site, then it will direct to the main directory for example, http://www.your-domain.com/robots.txt and if search engines did not find it in the root directory, then will conclude that your site doesn’t have a robot.txt file, then search engines will index all of which discovered during crawling your site and don’t be surprised if you see the entire contents of the site and your confidential files indexed and displayed on the search engines.

FREE Cloud Storage