DEV Community

a.infosecflavour
a.infosecflavour

Posted on

Jr Penetration Tester- Content Discovery- robots.txt

Closing the door of the first room, we're going to Discover(y) the Content of the second room. 🚪
We found a paper containing a question: Task 2- What is the directory in the robots.txt that isn't allowed to be viewed by web crawlers?

Accessing http://_machine_IP/robots.txt_, the message below shows up:

message

We're writing down the answer ✍️ /staff-portal. Just out of curiosity, let's see what's behind http://machine_IP/staff-portal. Here's the result:

robots

Remember the very first exercise of Intro to Offensive Security? We used gobuster to retrieve the hidden pages. Try to use the same command, to see if robots.txt can be found.

Top comments (0)