What it reads
Only addresses that appeared in the question it was asked, in a search result or on a page it has already read. It does not crawl sites.
- Company websites: what a company sells, where and to whom.
- Public registers and public tenders.
- News and other public pages that a search pointed to.
What it never reads
- Social media and people's profiles, for example LinkedIn, Facebook or Instagram.
- Search engine result pages.
- Paywalled news and commercial credit registers: they are cited from the search result, never opened.
- Anything behind a login, a bot check or a robots.txt rule that keeps it out.
How it behaves
- It reads your robots.txt before a page, and keeps to the rules for ToimeenBot or for all bots. If the server answers robots.txt with an error, or does not answer, it leaves the site alone.
- It respects the X-Robots-Tag header: a page marked noai or none is not read.
- When a site shows a bot check, it stops and does not try again.
- At most 30 pages for one piece of research, one page at a time, and never more than 2 MB or 10 seconds per page.
- It follows at most five redirects, and only to public addresses.
How to block it
Add these lines to your robots.txt. ToimeenBot reads robots.txt again at least once an hour.
User-agent: ToimeenBot Disallow: /
Or keep it off single pages with this response header:
X-Robots-Tag: noai
What happens to what it reads
- A page is used only in the report for the company that asked, with your address shown as the source.
- The text of pages read is deleted after 180 days. Pages are not used to train AI models.
- It researches companies, markets and tenders, not private persons.
Contact
Questions about ToimeenBot, or a request to stop reading your site: write to us.
hello@toimeen.com