Our crawler
If you are here, it is because you saw this name in your server log. This is what it does, what it does not do, and how to shut it out.
What it is
GoodNich is a content operating system for niche publishers. When a publisher adds a source — their own feed, a public channel, an address someone passes on to us — we open that specific address to read the text and decide whether it is worth a story.
It does not crawl your site. It does not follow the links it finds inside, it does not go looking for new addresses, and it does not come back on its own: it opens the address it already knew, reads, and leaves.
How it identifies itself
It always arrives with this User-Agent header, undisguised and with a link to this very page inside it. If what you see in your log is not exactly this, it is not us:
GoodNichBot/1.0 (+https://goodnich.io/bot)
And this is the name you can address in your robots.txt:
GoodNichBot
How to shut it out
We read your robots.txt before every download and we do what it says. To close the whole site, these two lines are enough:
User-agent: GoodNichBot Disallow: /
You can also close off just part of it, with whatever path you want under Disallow, and leave the rest open with Allow. When two rules match, the longer pattern wins, and on a tie Allow wins: that is the usual convention, and we follow all of it.
When in doubt, it does not download. If your server does not answer the robots.txt, or answers with an error of its own, we treat that as a refusal and stop there. Only when the file does not exist do we take it that there is no ban, which is what the convention says.
What it does not do
- It does not fill in forms or open links that do something — unsubscribes, confirmations, single-use links: it discards them before touching them.
- It does not try to get around a paywall or a sign-in. If the page does not hand over the text as it is, it goes without.
- It does not take the page. It takes the article text, up to a length limit, and drops the rest: scripts, menus, ads and footers stay out.
- It does not republish your content. What comes out, if anything does, is a story of our own that credits you by name and links to you.
If you want to tell us something
robots.txt is the channel that works today: we read it before every download, and you need neither write to us nor wait for an answer.