• Welcome to Web Hosting Community Forum for Webmasters - Web hosting Forum.
 

Recommended Providers

Fully Managed WordPress Hosting
lc_banner_leadgen_3
Fully Managed WordPress Hosting

WordPress Theme

Divi WordPress Theme
WPZOOM

Forum Membership

Forum Membership

What is robots.txt? Does it have any value for SEO?

Started by richard branson, September 29, 2012, 12:46:38 PM

richard branson

it is used by Google to know what you do NOT want indexed. So if you have content on your website, you do NOT want crawled, indicate that in robots.txt. .


edword84

In my opinion Robots.txt is a text file that you put on your site to tell search robots which pages you would like them not to visit. It is used for privacy purpose.

mike223

If you don't like to crawl any pages of your website by Google then this file will help you.

babarihun12

Robot.txt is a text file which is reside in the root directory of a website. It is used to inform search engine robots about not to index some web pages of the website because of security purpose. While search engine robots comes to any website, it first checks this file before crawl any web page.

James Andy

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

sherlyroger86

Robot.txt plays an important role in SEO. It helps search engine to inform about the pages which are not to be crawled or indexed. It is suggested by the web owner. He puts paths of web pages in this file that he doesn't want to crawl.

JimyEdword

Quote from: sherlyroger86 on November 26, 2012, 06:16:15 PM
Robot.txt plays an important role in SEO. It helps search engine to inform about the pages which are not to be crawled or indexed. It is suggested by the web owner. He puts paths of web pages in this file that he doesn't want to crawl.

Exactly sherly! I'm totally agree with you. It is very helpful file for search engine as well as end users like you and me.
You can make sure about the submission in certain websites if they contains robot.txt file.   ;)

Steve Smith


Thechipper

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

RH-Calvin

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Cheapest VPS | $1 VPS Hosting | Unmetered Bandwidth

grofee

A robots.txt file is a set of instructions for bots. This file is included in the source files of most websites. Robots.txt files are mostly intended for managing the activities of good bots like web crawlers, since bad bots aren't likely to follow the instructions.