PCI Information Alerts related to robots.txt

[David M. Graham]

Changing file permissions is likely to break program functionality in many cases.  In ecommerce software, this is likely to cost the store owner money in lost sales and can certainly HURT security as much as it helps.  Render my debug files read only, and I will move to a new host.

What I had in mind above is that you could use robots.txt to indicate which folders should be monitored for modifications - and cases in which a good opportunity to communicate with clients about security issues may exist.

David


 

[inetbiz]

Second - for administrators who are experienced with particular web applications they can indicate areas of misdirected concern on the part of the site operator.  This presents opportunities for active cooperation in building more secure sites.

This kind of issue is leading towards a newer model of distributed security rather than the antiquated and inadequate idea that perimeter security for a server is all that is important.

It still is still important, of course - but it is not possible to have perfect server perimieter security and a internet commerce.  The optimal solution is a collaborative one which addresses the full range of security concerns and works constantly to build a trustworthy web.

Done exactly that. Cpanel has a security thread that I've began to ask collaboration upon. You can visit that at forums.cpanel.net - Change owner of files thread.

The bash script will use bash find and bash xargs and I really want to extend upon many open source project folder checking and change world execute,write on some of the underlying folders that robots may uncover for hacker scanning runs.

[David M. Graham]

You can't.

Obviously if a good web spider can read robots.txt so can a bad one - and vice versa. Spoofing of ID is just not that hard.  This is a "Catch 22" vulnerability which cuts two ways.

Remove the information and it becomes indexable and will be advertised FOR YOU by various search engines.  Leave it and the information helps crackers locate good places to drive wedges into your system.

Funny thing is, the information can also help system administrators in a couple of ways - depending on their level of familiarity with the web applications served from their host machines.

First - the presence of these "no go " zones can help give the administrator information about unfamiliar applications, presuming that either the user (or the developer in the case of a distributed default robots.txt)  is less ignorant of the applications weak points.   

Second - for administrators who are experienced with particular web applications they can indicate areas of misdirected concern on the part of the site operator.  This presents opportunities for active cooperation in building more secure sites.

This kind of issue is leading towards a newer model of distributed security rather than the antiquated and inadequate idea that perimeter security for a server is all that is important.

It still is still important, of course - but it is not possible to have perfect server perimeter security and a internet commerce.  The optimal solution is a collaborative one which addresses the full range of security concerns and works constantly to build a trustworthy web.

[inetbiz]

Some Web servers use a file called /robot(s).txt to make search engines and any other indexing tools visit their Web pages more frequently and more efficiently.

By connecting to the server and requesting the /robot(s).txt file, an attacker may gain additional information about the system they are attacking. Such information as: restricted directories, hidden directories, CGI script directories, etc. may be available.

Take special care not to tell the robots not to index sensitive directories, since this tells attackers exactly which of your directories are sensitive.

So if McAfee reports a level 1 information vulnerability, how can a server administrator and developer protect, yet allow usage of a robots.txt file?