Bethesda's epic sci-fi RPG is here, and it's a big one. From shipbuilding to exploring the surface of Mars, our thoughts so far.
Starfield Review... In Progress
The first trailer for Grand Theft Auto 6 is finally here.
Grand Theft Auto 6 Trailer
We take an in-depth look at Avatar: Frontiers of Pandora and tell you why it should be heavily on your radar!
Avatar: Frontiers of Pandora - a Deep-Dive into its Potential
Range-wise, the ROG Rapture GT6 is phenomenal, and it's ideal for all gaming and non-gaming-related tasks.
ASUS ROG Rapture GT6 WiFi 6 Mesh System Review
FlashGet Downloads Blocked
Post by trog @ 01:13pm 03/12/09 | 32 Comments
We have made the decision to temporarily block the popular download manager FlashGet from working on files downloaded from the AusGamers Files Library.

FlashGet does this thing where it ignores errors and repeatedly hammers the servers over and over again. This is annoying because it makes the servers do more work than they need to be doing.

If you use FlashGet to try to download you'll notice you just get redirected to an image file that includes a link to this post. We'll have more information soon.


ausgamers





Latest Comments
  Link  
trog
Posted 02:47pm 03/12/09
If there is anyone out there with FlashGet that wants to help me test it out that would be helpful! If you try a download off the AG file section you should just end up with a simple .jpg file.
  Link  
natslovR
Posted 03:45pm 03/12/09
Is there a guide coming on polite settings for flashget?
  Link  
tequila
Posted 03:47pm 03/12/09
couldn't you just limit the number of apache connections per host?
  Link  
FraktuRe
Posted 03:57pm 03/12/09
See explanation here. :)
  Link  
re so
Posted 04:00pm 03/12/09
FlashGet rules!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  Link  
groganus
Posted 04:03pm 03/12/09
People read trogs blog... geez, youd figure the amount of whineing he does on the forums is enough to turn anyone off wanting to read anything else he writes.
  Link  
3x0dus
Posted 04:04pm 03/12/09
till the flashget users search google to change the user agent string.

changing the connection per host dosnt help becuase they are still doing x amount of repeated/uneeded requests ignoring 403a/404.

Can you not change apache via mod rewrite or similar so there are no 403/404 responses and all actualy goto a dummyfile/webpage?


  Link  
re so
Posted 04:05pm 03/12/09
Easiest fix would've been just allowing 5 connections per IP right? :P
  Link  
trog
Posted 04:13pm 03/12/09
Is there a guide coming on polite settings for flashget?
I don't think the settings can be set to make it not suck, unfortunately.

My first priority is to find a better download manager that works for everyone in a non-sucky way. So far all I have is WackGet but apparently it doesn't work on Vista so it's not an ideal solution.
till the flashget users search google to change the user agent string.
Well, we'll still catch them doing it and they'll just get IP blocked or something.
Easiest fix would've been just allowing 5 connections per IP right? :P
Yeh this is probably what we will end up doing eventually.
  Link  
mongie
Posted 04:18pm 03/12/09
Downthemall for Firefox.
  Link  
trog
Posted 04:25pm 03/12/09
Downthemall for Firefox.
DownThemAll unfortunately doesn't let you change your download URL, so it doesn't work with s***** systems like ours that give you a new URL each time if your old one fails for whatever reason.

We're looking to change the entire back end system to minimise that eventually though.
  Link  
Jim
Posted 05:07pm 03/12/09
Easiest fix would've been just allowing 5 connections per IP right? :P

Yeh this is probably what we will end up doing eventually.


I thought we already limited the number of connections per client to 4 or something

and unless we're serving some fancy parsed page for 403/404 what's the difference between that and a 5xx?


or when you say that's what we'll probably end up doing, are you talking about counting their tcp sessions and blocking them there?
  Link  
Finno
Posted 05:11pm 03/12/09
yeah I just tried flashget and it only D/Ls the image you mentioned.

I only use flashget at work to keep my rate low so I don't lag the other users (idiots won't upgrade). I don't use FG at home, where I do my AG downloading. So don't blame me!
  Link  
Midda
Posted 05:11pm 03/12/09
I wish Google Chrome had DownThemAll. It's the only thing I miss from Firefox.
  Link  
trog
Posted 06:11pm 03/12/09
I thought we already limited the number of connections per client to 4 or something

and unless we're serving some fancy parsed page for 403/404 what's the difference between that and a 5xx?

or when you say that's what we'll probably end up doing, are you talking about counting their tcp sessions and blocking them there?
we limit to 4, yeh - but flashget tries to make 5.

But I've just remembered how it works and changing to 5 won't help, because it just keeps making subsequent connections. So what happens is, through a file download, the link expires, then flashget tries to make a new connection. The link has expired, so it gets a 404. But Flashget doesn't believe in 404s, so it just keeps trying, like 5 times a second, to get the file, forever until the user realises and closes it, which apparently never happens.

This change has already had some effect, but there's still some FlashGet clients that must be really old or s*** or something that don't follow redirects because I'm still seeing spamming :( I HATE IT SO MUCH!@#
  Link  
3x0dus
Posted 06:37pm 03/12/09
what if you allow that useragent only to be given a new hashed download url rather then the 404/403.

that way they *do* Finish the file, and stop sending the crappy requests.
  Link  
trog
Posted 06:38pm 03/12/09
what if you allow that useragent only to be given a new hashed download url rather then the 404/403.

that way they *do* Finish the file, and stop sending the crappy requests.
at the moment, only mostly my blind rage at FlashGet and my desire to take it out on the users of it.

More advanced options are on the cards though; this is just a temporary change to see what effect it has on the network/servers.
  Link  
Jim
Posted 09:52pm 03/12/09
aside from log writes and disk space, prolly not a great deal
we could just stop logging those fails
  Link  
Oracle
Posted 10:35pm 03/12/09
Free Download Manager.
Open source, free, and works with FlashGot on Firefox.
http://www.freedownloadmanager.org/
http://flashgot.net/
  Link  
trog
Posted 10:45pm 03/12/09
FDM is what I was going to recommend, but more investigation is required - it seems they have been not releasing the source for the latest versions, so I'm a little concerned there. Also I can't find the source for FDM Lite.

I actually downloaded the source for the old version with a view to seeing if it could be simplified (and have considered their custom branded version even), but it's a bit of a project; I'm looking at other alternatives first.
  Link  
whoop
Posted 11:28pm 03/12/09
First thing I do when I install flashget is set it to 1 connection per server & 1 section. Seriously, who needs more than that these days?
  Link  
tequila
Posted 11:31pm 03/12/09
what's the point of that ^ ?
the idea of FlashGet is to speed up the download no?
All you're using it for with 1 connection is resume-support, which is built into every browser I've used for the last year or 3
  Link  
whoop
Posted 11:36pm 03/12/09
To stop it getting my ip banned from servers that ban your IP if you hammer them? And to stop places like, oh Ausgamers, from having to block things like, oh FLASHGET HAMMERING THE SERVERS.
  Link  
tequila
Posted 11:39pm 03/12/09
uh ok I think you missed my point
you can still stop it attacking servers by using 2-3 connections per server

but by using only 1 connection you basically just get resume support over a standard browser download manager, yes?
my point being, browsers these days have resume (at least ff3 does, surely ie-whatever-is-latest does?)
  Link  
trog
Posted 10:21am 04/12/09
you can still stop it attacking servers by using 2-3 connections per server
nah that's really one of my gripes - by default FlashGet a) doesn't respect Retry-After headers for 503 HTTP error codes and b) handles other error conditions really really stupidly. Continuing to hammer servers when you get a 404 seems utterly, utterly retarded to me. File is not there, what possible reason is there to think it's going to come back? And if you're going to try to see if it's back, don't test 5 times a second every second for infinity amounts of time, just check once every 30 seconds and stop after 5 minutes or something.

It really just needs some sane defaults to stop it sucking so much.
  Link  
natslovR
Posted 11:52am 04/12/09
i set mine to 5 sections per file, max of 5 connections per server, 0 retries

well, i did, i don't use it any more.

Would that stop your problems? if so, showing how it's done would be better than forcing people to use another program when they are happy with the one they have.
  Link  
trog
Posted 12:50pm 04/12/09
Would that stop your problems? if so, showing how it's done would be better than forcing people to use another program when they are happy with the one they have.
As above, it totally depends on some low-level details about how it works when it makes new connections.

Definitely don't want people to change what they're using if they don't have to, especially if I can't find a better alternative to recommend!

If I can identify more of the broken downloaders I'll try and contact the developers to find out why they're doing what they're doing.
  Link  
Oracle
Posted 07:39pm 04/12/09
"FDM is what I was going to recommend, but more investigation is required - it seems they have been not releasing the source for the latest versions, so I'm a little concerned there. Also I can't find the source for FDM Lite"


The sources are available from the sourceforge page, http://sourceforge.net/projects/freedownload/files/
They appear to go all the way up to the latest version, 3.0.848 unless there's something I'm missing.
FDM Lite I don't know about, it seems like it's the same thing with the modules available as separate downloads, so perhaps it's the same source just compiled modularly. TBH I've been using it for years but haven't updated it since 2.1.454
  Link  
trog
Posted 08:19pm 04/12/09
The sources are available from the sourceforge page, http://sourceforge.net/projects/freedownload/files/
heh they are now, I must've looked before the 25th of Nov :)

Yeh not sure about FDM Lite either; I need to take a closer look at the source. But the "Lite" version isn't light enough for what I want anyway
  Link  
whoop
Posted 08:53pm 04/12/09
FDM is worse for connections than flashget, on default settings I get "too many connections from your IP" errors from either telstra or GA.
  Link  
Oracle
Posted 10:37pm 04/12/09
I believe the default is to establish 4 connections per server. 4-5 is the default for most download managers. Some servers, as in your case, refuse more than one connection at a time. This isn't the fault of the download software, they're doing as they're meant to with sensible default settings. You'd get the same issue with any download manager, except for apparently FlashGet, which will just keep trying the connection forever.

FDM allows you to change the default connections for all downloads, or for individual downloads. The same can be applied for pause between connection attempts, maximum number of connection attempts, connection timeouts, bandwidth limiting etc. Also, if a connection is refused (eg because the server rejects more than x connections), it won't retry that connection at all from my experience.

It also supports download mirrors and changing the URL mid-download.
And no, I don't work for FDM.
  Link  
Jim
Posted 11:06pm 04/12/09
we don't refuse more than one connection at a time, at least as far as I can tell by looking at the nginx config files - one instance has 5 as the limit and the other has 4

Commenting has been locked for this item.
32 Comments
Show
Follow Us
Advertisement
Latest Threads
Log In
User:  
Pass:  

Features

Checking Out the Coolest PC Gaming Hardware at PAX Australia 2023

Reviews

ASUS ROG Rapture GT6 WiFi 6 Mesh System