[Blabber] I should not have said throttled

Eugene Lee leee at mit.edu
Mon Nov 30 13:27:06 UTC 2015


What you're doing most likely violates the site's ToS. In other words,
you're still doing it wrong.

Perhaps there's a very good reason why these sites that you
are"...accessing blocks large cloud servers as well as the major proxy
services (instantproxies, ezproxies, etc)."

If you intend on being a bad citizen on the net, then go find a
bulletproof host - they're the ones that offer VPSes and other
services at prices that make you go "why wouldn't their current
clients just use DO or AWS?" Chances are, they're running machines for
clients who require low latency networks to perform transactions for
financial gain.

--leee


On Mon, Nov 30, 2015 at 7:50 AM, Sherard Hall <smhall05 at gmail.com> wrote:
> I should not have said throttled, since that is not what is happening. The
> server that I am accessing blocks large cloud servers as well as the major
> proxy services (instantproxies, ezproxies, etc). The work I am doing
> requires that I quickly access the site to make a purchase. Once the
> purchase is complete, I can shutdown until the next item I am interested in
> releases.
>
> On Mon, Nov 30, 2015 at 7:00 AM, <blabber-request at list.hackmanhattan.com>
> wrote:
>>
>> Send Blabber mailing list submissions to
>>         blabber at list.hackmanhattan.com
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         https://list.hackmanhattan.com/listinfo/blabber
>> or, via email, send a message with subject or body 'help' to
>>         blabber-request at list.hackmanhattan.com
>>
>> You can reach the person managing the list at
>>         blabber-owner at list.hackmanhattan.com
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Blabber digest..."
>>
>>
>> Today's Topics:
>>
>>    1. Looking for Private Proxies or local datacenter to run    web
>>       crawler (Sherard Hall)
>>    2. Re: Looking for Private Proxies or local datacenter to run
>>       web crawler (Eugene Lee)
>>    3. Re: Looking for Private Proxies or local datacenter to run
>>       web crawler (Robby O'Connor)
>>    4. Re: Looking for Private Proxies or local datacenter to run
>>       web crawler (Barry Anderson)
>>    5. Kickstarter for my Aaron Swartz SecureDrop/Whistleblower
>>       Movie (Lisa Rein)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Sun, 29 Nov 2015 12:09:54 -0500
>> From: Sherard Hall <smhall05 at gmail.com>
>> To: blabber at list.hackmanhattan.com
>> Subject: [Blabber] Looking for Private Proxies or local datacenter to
>>         run     web crawler
>> Message-ID:
>>
>> <CAH=v_k=UQAWT+cWQH3gtXdfP7TTaHakZ4c6s03G0NDsSwwsrzg at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Hello everyone,
>>
>> I am looking for someone that can provide private proxies or a small scale
>> data center to run a web crawler. AWS, DigitalOcean, etc are not desirable
>> since their IP ranges are well known by most sites and throttled/blocked.
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> <https://list.hackmanhattan.com/pipermail/blabber/attachments/20151129/7d88fde3/attachment-0001.html>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Sun, 29 Nov 2015 16:25:42 -0500
>> From: Eugene Lee <leee at mit.edu>
>> To: "Hack Manhattan!" <blabber at list.hackmanhattan.com>
>> Subject: Re: [Blabber] Looking for Private Proxies or local datacenter
>>         to run web crawler
>> Message-ID:
>>
>> <CABnD-4Wg6UQD707sEmaK83DwE5eaF7Th9B7W=J+GvNZwHnSJ1A at mail.gmail.com>
>> Content-Type: text/plain; charset=UTF-8
>>
>> What's the context?
>>
>> If you're being throttled, you're either doing it wrong or you're
>> running a abusive crawler (which is still wrong).
>>
>> --leee
>>
>>
>> On Sun, Nov 29, 2015 at 12:09 PM, Sherard Hall <smhall05 at gmail.com> wrote:
>> > Hello everyone,
>> >
>> > I am looking for someone that can provide private proxies or a small
>> > scale
>> > data center to run a web crawler. AWS, DigitalOcean, etc are not
>> > desirable
>> > since their IP ranges are well known by most sites and
>> > throttled/blocked.
>> >
>> > _______________________________________________
>> > Blabber mailing list Blabber at list.hackmanhattan.com
>> > https://list.hackmanhattan.com/listinfo/blabber
>>
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Sun, 29 Nov 2015 20:27:27 -0500
>> From: Robby O'Connor <robby.oconnor at gmail.com>
>> To: Hack Manhattan! <blabber at list.hackmanhattan.com>
>> Subject: Re: [Blabber] Looking for Private Proxies or local datacenter
>>         to run web crawler
>> Message-ID: <565BA5FF.8030106 at gmail.com>
>> Content-Type: text/plain; charset=utf-8
>>
>> Yeah, what the hell are you doing that you're getting throttled?
>>
>> On 11/29/2015 04:25 PM, Eugene Lee wrote:
>> > What's the context?
>> >
>> > If you're being throttled, you're either doing it wrong or you're
>> > running a abusive crawler (which is still wrong).
>> >
>> > --leee
>> >
>> >
>> > On Sun, Nov 29, 2015 at 12:09 PM, Sherard Hall <smhall05 at gmail.com>
>> > wrote:
>> >> Hello everyone,
>> >>
>> >> I am looking for someone that can provide private proxies or a small
>> >> scale
>> >> data center to run a web crawler. AWS, DigitalOcean, etc are not
>> >> desirable
>> >> since their IP ranges are well known by most sites and
>> >> throttled/blocked.
>> >>
>> >> _______________________________________________
>> >> Blabber mailing list Blabber at list.hackmanhattan.com
>> >> https://list.hackmanhattan.com/listinfo/blabber
>> > _______________________________________________
>> > Blabber mailing list Blabber at list.hackmanhattan.com
>> > https://list.hackmanhattan.com/listinfo/blabber
>> >
>>
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Sun, 29 Nov 2015 21:31:56 -0500
>> From: Barry Anderson <grandbander at gmail.com>
>> To: "Hack Manhattan!" <blabber at list.hackmanhattan.com>
>> Subject: Re: [Blabber] Looking for Private Proxies or local datacenter
>>         to run web crawler
>> Message-ID:
>>
>> <CAAATaeGKjcTEB39Wc81OGhFsSJwdPaJtwkwBXQzN_SE0BnzAnw at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> I've recently used AWS to download a few 100 GB of pictures and never
>> encountered throttling on any website.  As long as you are not hitting
>> websites in a ridiculously redundant way you would have to do a lot to get
>> throttled.  TWC is another story when it comes to throttling, thus why I
>> had to move to AWS.  Alas, getting the data out of AWS was expensive.
>>
>>
>>
>> On Sun, Nov 29, 2015 at 8:27 PM, Robby O'Connor <robby.oconnor at gmail.com>
>> wrote:
>>
>> > Yeah, what the hell are you doing that you're getting throttled?
>> >
>> > On 11/29/2015 04:25 PM, Eugene Lee wrote:
>> > > What's the context?
>> > >
>> > > If you're being throttled, you're either doing it wrong or you're
>> > > running a abusive crawler (which is still wrong).
>> > >
>> > > --leee
>> > >
>> > >
>> > > On Sun, Nov 29, 2015 at 12:09 PM, Sherard Hall <smhall05 at gmail.com>
>> > wrote:
>> > >> Hello everyone,
>> > >>
>> > >> I am looking for someone that can provide private proxies or a small
>> > scale
>> > >> data center to run a web crawler. AWS, DigitalOcean, etc are not
>> > desirable
>> > >> since their IP ranges are well known by most sites and
>> > throttled/blocked.
>> > >>
>> > >> _______________________________________________
>> > >> Blabber mailing list Blabber at list.hackmanhattan.com
>> > >> https://list.hackmanhattan.com/listinfo/blabber
>> > > _______________________________________________
>> > > Blabber mailing list Blabber at list.hackmanhattan.com
>> > > https://list.hackmanhattan.com/listinfo/blabber
>> > >
>> > _______________________________________________
>> > Blabber mailing list Blabber at list.hackmanhattan.com
>> > https://list.hackmanhattan.com/listinfo/blabber
>> >
>>
>>
>>
>> --
>> Barry Anderson
>> cell 917-922-4823
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> <https://list.hackmanhattan.com/pipermail/blabber/attachments/20151129/3734d58e/attachment-0001.html>
>>
>> ------------------------------
>>
>> Message: 5
>> Date: Sun, 29 Nov 2015 18:37:25 -0800
>> From: "Lisa Rein" <lisa at lisarein.com>
>> To: blabber at list.hackmanhattan.com
>> Subject: [Blabber] Kickstarter for my Aaron Swartz
>>         SecureDrop/Whistleblower        Movie
>> Message-ID: <fdddb7d75df742ae43ff8cb26228de71.squirrel at mail.ctyme.com>
>> Content-Type: text/plain;charset=iso-8859-1
>>
>> Hi Hackmanhattan :-)
>>
>> I wanted to let you know about my little Kickstarter campaign for a movie
>> I am producing independently, "From DeadDrop to SecureDrop" - it's about
>> "SecureDrop," an anonymous upload system for whistleblowers that Aaron
>> prototyped with Kevin Poulsen, one month before his death, that connects
>> directly to news organizations (such as the Washington Post, Forbes, and
>> The New Yorker):
>>
>> https://www.kickstarter.com/projects/261957173/from-deaddrop-to-securedrop
>>
>> or
>>
>> bit.ly/SDrop
>>
>> I'm including a description of the project below! It's only going on for 9
>> more days, and I only have about $4,000 of the $20,000 I'm trying to
>> raise, so any help would be greatly appreciated!
>>
>> Thanks! :)
>>
>> Much love,
>>
>> lisa
>>
>>
>>
>> The film is entitled "From DeadDrop to SecureDrop," and provides a
>> historical account of how the "SecureDrop" open source anonymous
>> whistleblowing submission platform was originally prototyped by Kevin
>> Poulsen and Aaron Swartz, and later, after Aaron's death, how it made its
>> way its current home at the Freedom of the Press Foundation.
>>
>> People in the film (so far) include: Wired's Kevin Poulsen (how he was
>> inspired by WikiLeaks and then convinced Aaron to work with him on the
>> project), Trevor Timm and John Perry Barlow (of the Freedom of the Press
>> Foundation), who sought out the DeadDrop prototype, Garrett Robinson
>> (SecureDrop's current Development Lead) and Bill Budington (an EFF
>> engineer who worked directly on the early codebase), and some influential
>> figures familiar with the issues, such as Brewster Kahle (Founder of the
>> Internet Archive and co-founder, with myself, of Aaron Swartz Day) and
>> Cindy Cohn (Executive Director of the Electronic Frontier Foundation). I
>> also interviewed Virgil Griffith (Tor2Web and Wikiscanner), as he and
>> Aaron were good friends and worked on Tor2Web together.
>>
>> Jacob Applebaum has also agreed to be interviewed for the film. (Besides
>> being a WikiLeaks volunteer, he was good friends with Aaron.)
>>
>> After learning more about whistleblower Chelsea Manning (in the course of
>> my preparing to read her statement during my live Aaron Swartz Day event)
>> video: https://youtu.be/qle9UvJGdYg?t=1h23m33s statement:
>> http://www.aaronswartzday.org/chelsea-manning-2015/)...
>>
>> I came to realize that the film really needs to expand beyond the subject
>> of the SecureDrop technology itself, to attempt to explain *why* these
>> whistleblowers might decide to use WikiLeaks or SecureDrop to expose
>> corruption in the first place: To inform the public and make the world a
>> better place. I realize this sounds simple and obvious, but I don't think
>> that most people are actually aware of this. Here's a post explaining how
>> Ed Snowden chose the Constitution over a Non-Disclosure agreement, when he
>> decided to blow the whistle: http://www.aaronswartzday.org/snowden-oath/
>>
>> A third theme of the movie deals directly with an observation made by more
>> than one of the people I interviewed for the film, about how SecureDrop
>> will "help the next Ed Snowden" communicate with Journalists safely (so no
>> one will have to go through what he went through, just to get a story
>> out).
>>
>> The film suggests that SecureDrop could potentially usher in a new
>> generation of whistleblower: One that won't necessarily have to put their
>> whole life at risk, in order to "do the right thing."
>>
>>
>> Lisa Rein
>>
>> http://www.aaronswartzday.org
>>
>>
>>
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> _______________________________________________
>> Blabber mailing list Blabber at list.hackmanhattan.com
>> https://list.hackmanhattan.com/listinfo/blabber
>>
>> ------------------------------
>>
>> End of Blabber Digest, Vol 52, Issue 23
>> ***************************************
>
>
>
> _______________________________________________
> Blabber mailing list Blabber at list.hackmanhattan.com
> https://list.hackmanhattan.com/listinfo/blabber


More information about the Blabber mailing list