Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Do you know hosting companies that allow crawling and scanning? Most explicitly forbid such things as it makes their ip addresses worth less.


To add on my other comment: Vultr and Linode are both providing guides on how to set up searx on their platforms, so there you have two that will not raise any issue.

https://www.vultr.com/docs/install-searx-with-nginx-on-ubunt...

https://www.linode.com/content/maximize-your-privacy-with-se...


Really? Past employers have done quite heavy automated web queries across several major providers, who never had an issue, even when discussing our use-cases with them. At one point we had ~1000 AWS EIPs in one account solely for increasing throughput (this was an effect of the legacy solution outgrowing its intended lifetime way longer than we wanted, if you scoff at how uneconomical that sounds and is. Just saying, AWS supported us throughout).

Either way, what we're talking about here is more akin to an http proxy than scanning/crawling - since every request is explicitly triggered by manual user action as opposed to automated so it shouldn't be any issue.

If you expose and advertise a public unauthenticated frontend and end up with 10ks of users maaaaybe some providers will start talking to you about it but otherwise I wouldn't have any concerns. And if you get to that scale, you may already want to look at proxying through providers like Luminati anyway.




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: