[Blabber] Looking for Private Proxies or local datacenter to run web crawler
grandbander at gmail.com
Mon Nov 30 02:31:56 UTC 2015
I've recently used AWS to download a few 100 GB of pictures and never
encountered throttling on any website. As long as you are not hitting
websites in a ridiculously redundant way you would have to do a lot to get
throttled. TWC is another story when it comes to throttling, thus why I
had to move to AWS. Alas, getting the data out of AWS was expensive.
On Sun, Nov 29, 2015 at 8:27 PM, Robby O'Connor <robby.oconnor at gmail.com>
> Yeah, what the hell are you doing that you're getting throttled?
> On 11/29/2015 04:25 PM, Eugene Lee wrote:
> > What's the context?
> > If you're being throttled, you're either doing it wrong or you're
> > running a abusive crawler (which is still wrong).
> > --leee
> > On Sun, Nov 29, 2015 at 12:09 PM, Sherard Hall <smhall05 at gmail.com>
> >> Hello everyone,
> >> I am looking for someone that can provide private proxies or a small
> >> data center to run a web crawler. AWS, DigitalOcean, etc are not
> >> since their IP ranges are well known by most sites and
> >> _______________________________________________
> >> Blabber mailing list Blabber at list.hackmanhattan.com
> >> https://list.hackmanhattan.com/listinfo/blabber
> > _______________________________________________
> > Blabber mailing list Blabber at list.hackmanhattan.com
> > https://list.hackmanhattan.com/listinfo/blabber
> Blabber mailing list Blabber at list.hackmanhattan.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Blabber