That being said, it *is* a concerning amount of power sometimes, yeah. I, uh, did accidentally fuck over some server admins once. (Not a full-on denial-of-service, but apparently they struggled pretty hard under the sudden spike in load.)
I've learned how to throttle since then ("--wait={{insert number of seconds here}}"), and I recommend you do the same, at least with large websites run by small-timers. (Unfortunately the things they did on their end to keep scraper bots from digging too deep (and ending up downloading one zillion immaterially-different page variants) don't seem to have worked, and I have still never successfully downloaded their site.)
(I...*guess* I could use my shiny new SingleFile and act as a manual web scraper? But the site is so big--even to an entity capable of seeing past the zillion variants--that that doesn't seem very feasible.)
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
timeasmymeasure
no subject
Date: 2020-04-10 01:43 am (UTC)---
That being said, it *is* a concerning amount of power sometimes, yeah. I, uh, did accidentally fuck over some server admins once. (Not a full-on denial-of-service, but apparently they struggled pretty hard under the sudden spike in load.)
I've learned how to throttle since then ("--wait={{insert number of seconds here}}"), and I recommend you do the same, at least with large websites run by small-timers. (Unfortunately the things they did on their end to keep scraper bots from digging too deep (and ending up downloading one zillion immaterially-different page variants) don't seem to have worked, and I have still never successfully downloaded their site.)
(I...*guess* I could use my shiny new SingleFile and act as a manual web scraper? But the site is so big--even to an entity capable of seeing past the zillion variants--that that doesn't seem very feasible.)