← DictionaryRobots.txt
also called robots file
A file instructing crawlers which parts of a site they may request.
A note telling crawlers where not to go.
What It Is
Robots.txt is part of Technology, and more precisely Search Tech. The canonical meaning is a file instructing crawlers which parts of a site they may request.
It is a mechanism rather than an opinion: set up properly it behaves predictably, and set up badly it fails just as predictably. Held that way, the term stays useful instead of turning into decoration.
How It's Used
Robots.txt earns its keep in build decisions, tooling and the plumbing behind a working system. In real work it sits close to Sitemap. The practical test is whether it is configured, documented and actually working.
Why It Matters
The cost of being vague about Robots.txt shows up later, not today. When it is unclear, small technical debts compound into a rebuild. Deciding it deliberately costs one conversation. Leaving it undecided costs several.
Think of a tool in a drawer — dull until the moment you need exactly it. A note telling crawlers where not to go.
Termshift™
How the meaning shifts depending on who is using it, and where.
Inside the industry, Robots.txt is used tightly and assumed to be understood: a file instructing crawlers which parts of a site they may request.
Away from the jargon, it is a note telling crawlers where not to go.
The Bottom Line
Robots.txt is not complicated — it is just often left vague. A note telling crawlers where not to go. Decide it, write it down, move on.