What is a robots.txt file ?

Well, a robots.txt file is actually not about those robots you read in science fiction, instead it deals with those script robots / bots that you read about in the previous post. As per the search engine crawler protocol, a robots.txt file is a file that you put on the directories of your site with instructions on how you want your site to be read by the bots (and not all the search engine crawlers obey it, but the major ones do). It is a very simple text file that you can create in Notepad (or any other text editor, so if you are a vi fanatic, then use that by all means). You can use this text file to even ask the bots to prevent certain sections of your site from being copied, which means that those sections of the site will not be visible in search engines (and why would you want this to happen ? Well, one example is when you use this set of instructions to prevent duplicate reading of your site, so for example, the you want all the archive section of your site to be read, but not other paths to the same parts of your site (to prevent the duplicate content penalty), then this is a good way to do this (and the All-In-One SEO Plugin does indeed do that)).

Some scenarios in which you want to use the robots.txt file

- It is a matter of good protocol to have a robots.txt file since search engines expect these files to be there (but this is not a necessary condition)
- If you are in the habit of building different doorway pages of a site, with customized versions of the doorway page for each search engine or platform (a different doorway page for the iPhone, for example), then you would want to customize the search engines entry into these doorways (for example, if the Google search bot found its way to all off these doorway pages, they might consider you to be trying to hoodwink users). Alert: Doorway pages are now under alerts by search engines and you should consider your policies and intent in this area carefully.
- if your page has many sections that are still under construction, then you would not want the search engines to crawl and show these not-ready sections of the site. In such cases, you can prevent the spiders from reading the site and showing the result in search engines
- There are some sections of your site that are meant for individual persons, and you are not really interested in making them available for wider reading. Other people can access these if they go through the whole structure of your site, but there is no reason to show the results in a search engine result.
- You have created a print friendly version of your articles, but really don’t want search engines to go there (to avoid the duplicate content penalty and also to prevent users from directly accessing the print friendly versions). So you can enforce this in the robots.txt file.
- Keep the robots away from such content as has no value for the bots, such as directories with flash content, with JavaScript, and so on.

This quick tutorial will provide you with an htaccess file that does the following:
1. Protects itself (security)
2. Turns the digital signature off (security)
3. Limits upload size (security)
4. Protects wp-config.php (security)
5. Gives access permission to all visitors with exceptions (security, usability)
6. Specifies custom error documents (usability)
7. Disables directory browsing (security)
8. Redirect old pages to new (optional)
9. Disables image hotlinking (bandwidth)
10. Enables PHP compression (bandwidth)
11. Sets the canonical or “standard” url for your site (seo, usability)

Once you get this done, you will have increased the security level of your site, as well as being able to handle visitors.

WordPress scanner (learn more at this link).
Running it is fairly simple, you have to move a plugin to your plugins directory and activate it. The plugin will add a link to your WordPress template. Once you are done, you need to disable the plugin (and be sure to do so). Once you have activated the plugin, you need to go to the wpscan page (link) and enter your blog details.

In addition to plugins, you need to evaluate the following.
- Check with your web hosts about whether directory browsing is allowed by default, if no index.html file is present. A lot of hosts have turned that off by default, but if it is on, then you should add an index.html file in your plugins directory; you don’t want people to know which plugins you are using
- Keep your WordPress installation updated, and one way to do the automatically is by using the “WordPress Automatic Upgrade plugin” (link) – Also, WordPress 2.7 onwards has an integrated update feature which you should use.
- Check for the security levels on your Forms / Comments input page. Use a more secure mailer for WordPress (Secure Form Mailer Plugin For WordPress)
- Don’t use plugins without reading a bit more about them. Plugins are made with the best of intentions, but it is quite possible that a plugin can lead to a security hole.

Admin SSL:
This plugin enhances the security level of the admin and wordpress login pages. Read more at this link.

- Supports All SSL Setups (Private and Shared)
- Encrypts cookie contents
- Compatible with all versions of PHP 4 and 5
- Easy to install (1 file uploaded)

Force SSL Plugin:
This plugin ensures a much higher level of security for your entire blog, forcing your users to use SSL. However, you will need to get a certificate for your blog, and that can be somewhat expensive, from a 3rd party certificate provider. This method ensures that your interaction between the client and the server is free of interception.