If you’ve seen EchelonGraphBot in your logs, here’s exactly what it does and how to block it

We run a crawler that checks how public websites are set up. Things like whether HTTPS is enforced, when the certificate expires, which TLS version is offered, what security headers are set, how cookies are flagged, and where redirects go. If it has been hitting your servers, you probably want to know what it is. So we wrote the whole thing down on one page: echelongraph.io/bot It covers how often it visits a single host, how it handles robots.txt and Retry-After, which address it comes from, what it will never do, and how to opt out with a DNS record. No signup, no popup. Two notes on the opt-out, since that is usually the first question. It is a DNS TXT record, so you do not need an account with us to use it. And robots.txt works the normal way if you would rather do it there. The data it collects is also public at echelongraph.io/radar if you want to look up your own domain, or anyone else’s. Free, no account. Things that are not finished, so you hear it from us first: – We say checks can run every 30 seconds. They actually land about once a minute. The scheduler does not honour the shorter setting yet. – Checks stop a couple of minutes after you close the tab. Happy to answer anything about the crawler’s behaviour. If it has done something in your logs that the bot page does not explain, tell me and I will fix either the bot or the page. submitted by /u/Foreign_Score_4021 [link] [comments]Technical Information Security Content & DiscussionRead More