Divi WordPress Theme

Topic: Robots.txt  (Read 2242 times)

Offline AradyaGD

  • New Member
  • *
  • Posts: 26
  • iTrader: +0/-0
    • View Profile
Robots.txt
« on: August 08, 2018, 06:58:57 PM »
Hi, I am new to SEO
I want to know about robots.txt, What is robots.txt?
VPS Hosting - AccuWebHosting

Offline JesicaJoh

  • Junior Guru
  • **
  • Posts: 61
  • iTrader: +0/-0
    • View Profile
Re: Robots.txt
« Reply #1 on: August 23, 2018, 07:25:14 AM »
It is the text file, You can represent your webpage allow or not allow for a bots or crawler.

Offline RH-Calvin

  • Junior Guru
  • **
  • Posts: 98
  • iTrader: +0/-0
    • View Profile
Re: Robots.txt
« Reply #2 on: August 27, 2018, 10:36:58 AM »
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

Offline marrysmith

  • Newbie
  • *
  • Posts: 5
  • iTrader: +0/-0
    • View Profile
Re: Robots.txt
« Reply #3 on: September 05, 2018, 05:36:40 AM »
With robots.txt you can tell search engine which directories to index and which to not, in simple words, when google bot comes to your website, it check robots.txt files, and follows the instructions to index the allowed directories.

Offline sankalppatil

  • Junior Guru
  • **
  • Posts: 153
  • iTrader: +0/-0
    • View Profile
Re: Robots.txt
« Reply #4 on: October 17, 2018, 11:30:16 AM »
Hello friends,
Robots.txt is a text file that lists web pages which contain instructions for search engines robots. The file lists web pages that are allowed and disallowed from search engine crawling.

Offline maheshseo

  • Junior Guru
  • **
  • Posts: 103
  • iTrader: +0/-0
    • View Profile
Re: Robots.txt
« Reply #5 on: February 26, 2020, 10:58:25 PM »
it's file to guide crawlers which page need to index

Offline Sagar789

  • New Member
  • *
  • Posts: 28
  • iTrader: +0/-0
    • View Profile
Re: Robots.txt
« Reply #6 on: May 13, 2020, 10:35:12 AM »
yes it's a text file used to allow or disallow particular directories  in a website
Shopify Divi WordPress Theme
Tags: