Software Index
Related Topics
Popular Trends
Trending Topics
 

Sample Robots Txt File

 All | Freeware
Product Title  /  Popularity Revised License Rating Size
Robots.txt
Robots.txt is a visual editor for Robot Exclusion Files and a log analyzer software. It allows a user to quickly and easily create the robots.txt files required to instruct search engine spiders, which parts of a Web site are not ...		 October 1st 2005 Shareware    2,870k
Advanced Robots.txt Generator
Create your robots.txt files in seconds and maintain them easilly and effortlesly with the unique utility Advanced Robots.txt Generator. The software supports 300 robots, works in a WYSWIWYG visual way and comes with full documentation and unlimited customer support. It's ...		 December 6th 2006 Shareware    1,577k
Oven Fresh Robots Txt Maker
Just fill in a simple form and the robots.txt code is generated for you. Robots Txt Maker will generate the robots.txt file you need to help prevent the indexing of files and directories that if indexed by themselves could frustrate ...		 February 19th 2011 Shareware    1,877k
RoboGen
RoboGen is a visual editor for Robot Exclusion Files. It allows the user to quickly and easily create the ROBOTS.TXT files required to instruct web search engines which parts of a web site are not to be indexed and made ...		 November 25th 2002 Shareware  4.5 stars 1,489k
Robots.txt editor
This program is a visual editor for Robot Exclusion Files and log analyzer software. It allows the user to quickly and easily create the robots.txt files required to instruct web search engines which parts of a web site are not ...		 February 22nd 2005 Shareware    2,869k
TracRobotsTxt
TracRobotsTxt is a module that will serve a robots.txt file from Trac. Mostly useful to tracd users, but works on anything. Just put the data you want in the wiki page RobotsTxt.

Configuration:

To enable the plugin:

[components]
robotstxt.* = enabled

A typical RobotsTxt wiki ...		 March 18th 2011 Freeware    10k
Page Promoter
Page Promoter is an ultimate URL registration, search engine optimization and promotion tool for submitting Web sites to thousands search engines and directories as well as preliminary site optimization and effectiveness check. The software allows you to enhance sites` exposure ...		 October 1st 2005 Shareware    7,078k
ASP Search Engine
Indexing of the title, description, keywords and full content of all HTML, ASP and TXT documents using a recursive call to all pages in a web site and sub folders excluding those folders listed in the robots.txt file. Simple search ...		 November 7th 2011 Shareware    97k
Oven Fresh Developer Pack
Make your robots.txt file and meta tags by filling in simple form fields.Now you may give your web site the edge it needs to achieve higher rankings within search engine databases. You get 2 programsfor one low price.Even calculate meta ...		 October 13th 2011 Shareware    4k
Website Toolbox Pro
Over 30 generators, Web 2.0 elements, tools and scripts for webmasters and web designers to speed up website building and save time and money. Website Toolbox Pro includes: a color palette generator, gradient images, stripe backgrounds, rounded corners, a favicon ...		 January 6th 2011 Shareware    283k
Oven Fresh PowerPack
Make an unlimited number of online voting polls, rollovers, popups, audio players, robots.txt files, and meta tags by filling in simple form fields.You get 6 programs for one low price. Best of all, no special servers, monthly fees, or additional ...		 August 17th 2011 Shareware    24k
1-abc.net Folder-To-TXT
Do you know this situation? You just burned a disk, you collected some data on a USB stick or you simply organized some files on your local hard drive. Now you want to get an editable list of all files ...		 June 5th 2008 Freeware  5 stars 303k
All to Text
All to Text V1.26 This program is well designed for convert files to txt file. All to Text can easily convert .html, .htm, .doc, .rft files to .txt file.
It can be especially useful if you require complicated file conversion ...		 June 28th 2006 Shareware    1,064k
1-abc.net File Finder
Select one or more paths or even a complete drive, configure how the searching process will behave and get a list of all these files you are missing. After that, you can save the results list as a *.txt file ...		 February 2nd 2007 Shareware    416k
VICalc
Simply open any of your txt files or use an empty new one. Enter 2 plus 3 and press Enter.
The result " = 5 " will be added automatically:
2 + 3 = 5

Use any of +, -, ...		 July 20th 2010 Demo    4,659k
Aiseesoft PDF to Text Converter
Aiseesoft PDF to Text Converter is best PDF Text Converter, which is designed to convert any PDF file to .txt format without the need of Adobe Acrobat or Adobe Reader software. With the help of PDF to TXT Converter, You ...		 August 1st 2011 Shareware    3,430k
FileName Pro
FileName Pro, a professional grade batch file naming utility. Whether you have a few or thousands of files to rename, FileName Pro can rename your files in seconds. Name files using any combination of file naming schemes. Name using ID3 ...		 April 21st 2006 Shareware  5 stars 4,225k
MS Word Extract Email Addresses From Documents Software
This application quickly searches Word files for email addresses and extracts them. Results can then be saved to a text file.		 July 1st 2006 Shareware    471k
Text Files to PDF Convert Software
Create PDF files from your many text files without Adobe products. No complicated options. Files are not required to have txt extensions.		 August 12th 2006 Shareware    4,219k
MySQL to Text Files Import, Export & Convert Software
Save MySQL tables to text or CSV files. In the resulting text file, each field (column) is delimited by a comma or other character, each row is delimited by a newline. You can choose which fields to save in the ...		 November 29th 2006 Shareware    448k
Related Scripts
Easy Robots.txt Generator
The robots.txt file is the first thing search engine "spiders" look for when indexing a website. The robots.txt file tells search engine spiders (robots) which files and/or directories they are NOT allowed to index. This helps to prevent incomplete site ...		 November 22nd 2005 Freeware     
robots.txt generator
The robots.txt generator is a script that is created using PHP programming with which you will be able to browse through the directory structure of your web server. And if a robots.txt is found, the corresponding folders are checked. In ...		 December 18th 2005 Freeware     
I, Robot
From this tutorial, you will gain the cost effective way to secure particular website pages from spiders, bots, crawlers, robots etc., Through the given program, you can learn how to denote the pages to be secured using the Disallow function ...		 December 4th 2005 Freeware     
JillyBoel Spambot Foiler
JillyBoel Spambot Foiler is a counter to Spambots that attempt to collect email addresses from your website. This is achieved by providing invalid email addresses to the spambot. This is carried out by creating and feeding invalid email addresses. This ...		 December 10th 2005 Freeware     
Web dev hosted tools
Tools include a link popularity tool, Robots.txt file generator, meta tag analyzer, screen resolution validator, .htaccess generator and more.		 November 25th 2005 Freeware     
robots.txt Tools And Tutorial
This site provides the tools to create and validate robots. It searches the engine spiders and similar for robots the main web directory. This is a plain text file that allows you to create or modify it with a text ...		 November 29th 2005 Freeware     
robots/text generator
robots/text generator is a multi-platform compatible script. It runs a check if robots.txt exists in your server's directory structure. If found, it is read and the appropriate allowed/disallowed folders are checked/unchecked.		 December 9th 2005 Freeware     
Unlimited XML Gallery
Features:
As the name says, it is unlimited in every way. You can add unlimited number of items in unlimited number of galleries. The scrollbar hides itself when it is not needed and it supports easing. Extensions like JPEG, PNG, GIF, ...		 May 21st 2009 Commercial    10k
ASPIP
This ASP Script logs the IP of everyone that passes through the page the script on to a TXT file. Though it is strongly suggested that you make an ASP Script that clears the IP Log every week if you ...		 November 29th 2005 Freeware     
Batch IP
This IP tool offers features such as convert all IP addresses of some domains which used muti-IP addresses, convert between IP address and domain/host name, ping IP address and find out which country the internet IP address belongs to. The ...		 December 18th 2005 Commercial     
Related Code
DB File writer
DB File Writer will get an input from a input.txt file containing results from DB SELECT query and modify the file to have INSERT commands with the values of input.txt and write it to output.txt file.

 March 30th 2006     5k
DSS Server Control 1.1 *Update*- Multi User Server Framework
Ever wanted to make your own multiple user server? Say for something like chat or Instant Messaging? Now you can forget about coding the Winsock control for complicated multi user support, let DSS Server Control do the job for you! ...		 August 31st 2006     62k
OutlineArt
OutlineART,applied in CAD wordprocessing,is a tool to transform words and symbols into vector-contour,we provide both windows desktop version and ARX embedded version for AutoCAD(supporting R14 and R2000),so that words and symbols can be transformed into not only DXF file,but also ...		 January 21st 2007     658k
Updating Minutes
Originally intended for Minutes for a meeting. You can read the description in the readme.txt file in the code. Must be able to make DSN's with your databases. Containes many aspects of ColdFusion, learn something!		 February 8th 2007     16k
DigiOz MySQL DLL
The DigiOz MySQL DLL is a DLL written in Visual Basic .NET 2005. This DLL allows you to Query, Insert, Update and Delete records in and out of a MySQL Database using the MySQL .NET Connector 1.0.7 or higher. The ...		 February 20th 2007     6k
New Software
Popular Software