• Welcome to Web Hosting Community Forum for Webmasters - Web hosting Forum.
 

Recommended Providers

Fully Managed WordPress Hosting
lc_banner_leadgen_3
Fully Managed WordPress Hosting

WordPress Theme

Divi WordPress Theme
WPZOOM

Forum Membership

Forum Membership

Robots.txt

Started by AradyaGD, August 08, 2018, 06:58:57 PM

AradyaGD

Hi, I am new to SEO
I want to know about robots.txt, What is robots.txt?

JesicaJoh

It is the text file, You can represent your webpage allow or not allow for a bots or crawler.

RH-Calvin

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Cheapest VPS | $1 VPS Hosting | Unmetered Bandwidth

marrysmith

With robots.txt you can tell search engine which directories to index and which to not, in simple words, when google bot comes to your website, it check robots.txt files, and follows the instructions to index the allowed directories.

sankalppatil

Hello friends,
Robots.txt is a text file that lists web pages which contain instructions for search engines robots. The file lists web pages that are allowed and disallowed from search engine crawling.

maheshseo

it's file to guide crawlers which page need to index

Sagar789

yes it's a text file used to allow or disallow particular directories  in a website

grofee

A robots.txt file is a set of instructions for bots. This file is included in the source files of most websites. Robots.txt files are mostly intended for managing the activities of good bots like web crawlers, since bad bots aren't likely to follow the instructions.