Web Hosting Forum - A Web Hosting community for Webmasters


Divi WordPress Theme

Author Topic: What is robots.txt? Does it have any value for SEO?  (Read 2122 times)

0 Members and 1 Guest are viewing this topic.

Offline richard branson

  • Junior Guru
  • **
  • Posts: 73
  • iTrader: +0/-0
    • View Profile
    • Intraday Tips
What is robots.txt? Does it have any value for SEO?
« on: September 29, 2012, 12:46:38 PM »
it is used by Google to know what you do NOT want indexed. So if you have content on your website, you do NOT want crawled, indicate that in robots.txt. .


VPS Hosting - AccuWebHosting

Offline edword84

  • New Member
  • *
  • Posts: 22
  • iTrader: +0/-0
    • View Profile
    • Xtremeseopackages
Re: What is robots.txt? Does it have any value for SEO?
« Reply #1 on: October 15, 2012, 09:48:39 AM »
In my opinion Robots.txt is a text file that you put on your site to tell search robots which pages you would like them not to visit. It is used for privacy purpose.

Offline mike223

  • Newbie
  • *
  • Posts: 2
  • iTrader: +0/-0
    • View Profile
    • Business Broadband Providers
Re: What is robots.txt? Does it have any value for SEO?
« Reply #2 on: October 16, 2012, 03:47:43 PM »
If you don't like to crawl any pages of your website by Google then this file will help you.

Offline babarihun12

  • Newbie
  • *
  • Posts: 9
  • iTrader: +0/-0
    • View Profile
    • Affordable Lawyer
Re: What is robots.txt? Does it have any value for SEO?
« Reply #3 on: November 08, 2012, 10:05:33 AM »
Robot.txt is a text file which is reside in the root directory of a website. It is used to inform search engine robots about not to index some web pages of the website because of security purpose. While search engine robots comes to any website, it first checks this file before crawl any web page.

Offline James Andy

  • Junior Guru
  • **
  • Posts: 57
  • iTrader: +0/-0
    • View Profile
    • Software Development Company Malaysia
Re: What is robots.txt? Does it have any value for SEO?
« Reply #4 on: November 19, 2012, 04:53:53 PM »
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
The best iPhone app Development and Android app Development Company in Malaysia..!

Offline sherlyroger86

  • Newbie
  • *
  • Posts: 7
  • iTrader: +0/-0
  • MG.net
    • View Profile
    • Mineral Gallery
Re: What is robots.txt? Does it have any value for SEO?
« Reply #5 on: November 26, 2012, 06:16:15 PM »
Robot.txt plays an important role in SEO. It helps search engine to inform about the pages which are not to be crawled or indexed. It is suggested by the web owner. He puts paths of web pages in this file that he doesn't want to crawl.

Offline JimyEdword

  • New Member
  • *
  • Posts: 13
  • iTrader: +0/-0
  • Newfoundland Driveway Sealing
    • View Profile
    • Newfoundland Line Painting
Re: What is robots.txt? Does it have any value for SEO?
« Reply #6 on: December 04, 2012, 06:16:49 PM »
Robot.txt plays an important role in SEO. It helps search engine to inform about the pages which are not to be crawled or indexed. It is suggested by the web owner. He puts paths of web pages in this file that he doesn't want to crawl.

Exactly sherly! I'm totally agree with you. It is very helpful file for search engine as well as end users like you and me.
You can make sure about the submission in certain websites if they contains robot.txt file.   ;)

Offline Steve Smith

  • Junior Guru
  • **
  • Posts: 233
  • iTrader: +0/-0
    • View Profile
Re: What is robots.txt? Does it have any value for SEO?
« Reply #7 on: April 08, 2013, 11:57:37 AM »
Thank you so much for this information..

Offline Thechipper

  • Junior Guru
  • **
  • Posts: 150
  • iTrader: +0/-0
    • View Profile
Re: What is robots.txt? Does it have any value for SEO?
« Reply #8 on: September 05, 2015, 04:41:43 PM »
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

Offline RH-Calvin

  • New Member
  • *
  • Posts: 29
  • iTrader: +0/-0
    • View Profile
    • Reprise Hosting
Re: What is robots.txt? Does it have any value for SEO?
« Reply #9 on: June 26, 2017, 05:23:03 PM »
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Cheapest VPS | $1 VPS Hosting | Unmetered Bandwidth

Divi WordPress Theme
Tags: