[Blabber] Looking for Private Proxies or local datacenter to run web crawler

Barry Anderson grandbander at gmail.com
Mon Nov 30 02:31:56 UTC 2015


I've recently used AWS to download a few 100 GB of pictures and never
encountered throttling on any website.  As long as you are not hitting
websites in a ridiculously redundant way you would have to do a lot to get
throttled.  TWC is another story when it comes to throttling, thus why I
had to move to AWS.  Alas, getting the data out of AWS was expensive.



On Sun, Nov 29, 2015 at 8:27 PM, Robby O'Connor <robby.oconnor at gmail.com>
wrote:

> Yeah, what the hell are you doing that you're getting throttled?
>
> On 11/29/2015 04:25 PM, Eugene Lee wrote:
> > What's the context?
> >
> > If you're being throttled, you're either doing it wrong or you're
> > running a abusive crawler (which is still wrong).
> >
> > --leee
> >
> >
> > On Sun, Nov 29, 2015 at 12:09 PM, Sherard Hall <smhall05 at gmail.com>
> wrote:
> >> Hello everyone,
> >>
> >> I am looking for someone that can provide private proxies or a small
> scale
> >> data center to run a web crawler. AWS, DigitalOcean, etc are not
> desirable
> >> since their IP ranges are well known by most sites and
> throttled/blocked.
> >>
> >> _______________________________________________
> >> Blabber mailing list Blabber at list.hackmanhattan.com
> >> https://list.hackmanhattan.com/listinfo/blabber
> > _______________________________________________
> > Blabber mailing list Blabber at list.hackmanhattan.com
> > https://list.hackmanhattan.com/listinfo/blabber
> >
> _______________________________________________
> Blabber mailing list Blabber at list.hackmanhattan.com
> https://list.hackmanhattan.com/listinfo/blabber
>



-- 
Barry Anderson
cell 917-922-4823
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://list.hackmanhattan.com/pipermail/blabber/attachments/20151129/3734d58e/attachment.html>


More information about the Blabber mailing list