Bethesda's epic sci-fi RPG is here, and it's a big one. From shipbuilding to exploring the surface of Mars, our thoughts so far.
Starfield Review... In Progress
The first trailer for Grand Theft Auto 6 is finally here.
Grand Theft Auto 6 Trailer
We take an in-depth look at Avatar: Frontiers of Pandora and tell you why it should be heavily on your radar!
Avatar: Frontiers of Pandora - a Deep-Dive into its Potential
Range-wise, the ROG Rapture GT6 is phenomenal, and it's ideal for all gaming and non-gaming-related tasks.
ASUS ROG Rapture GT6 WiFi 6 Mesh System Review
Server disconnects every single day. Why isn't this fixed alrea...
Predict
Melbourne, Victoria
5 posts
I've mentioned this in the forums, i've mentioned this in support tickets, but I'm simply being ignored. Both of our BF4 servers, each day will disconnect all players in the server at any random time, whether the server actually crashes (Full restart) only occurs in very rare cases, but the player disconnects is starting to really tick me off. My servers seem to spend more time losing players than gaining them due to this poor service. If you think im talking crap, please look at my statistics logs in our game create control panel if need be.

Can you guys do something about it or not?
12:11pm 09/02/14 Permalink
system
Internet
--
12:11pm 09/02/14 Permalink
Malthius
Brisbane, Queensland
1197 posts
I've just reviewed the log files for one of your servers for the last week.

I classify events where groups of players are removed from the server into two types, that I've very scientifically named "dumps" and "drops".

A "drop" is where the BF4 server loses connection to the back end ("drops" the back end connection) and reverts into a pre-game mode, disconnecting every player and staying offline for up to 2 minutes after the connection is restored.

Your server had 9 "drops" in the last week, but none of those removed any players, they were all at times when there was nobody online (e.g. 2014-02-08T14:22:38 or 2014-02-09T02:29:31).

A "dump" is where the BF4 server stays online and connected to the back end, but a cluster of players are removed from the server by the back end. (players are "dumped"). The server doesn't report any problem, it just sees a message from the back-end (BattleLog) to remove player #2, then player #3, then player #5 etc, all in within a few seconds.

Your server had 7 "dumps" in the last week, though most of them were earlier in the week and didn't kill the server - e.g. at 2014-02-04T21:58:27 23 of the 57 players were removed.

I'm suspect the one that caused you to make this post was 2014-02-09T11:56:15 where 59 of the 64 players were dumped.

----------

The reason this hasn't been "fixed" is that it is entirely outside our control - it appears to be a network problem affecting the back end servers. DICE are working on the problem and have promised us both a short term improvement and a long term permanent fix, but I don't have any ETA on either of those things. I have sent the log files for another server that is behaving like yours to DICE, and I'll hound them again for an answer later tonight (when they get back to work, it is still Sunday evening over there).
12:02pm 10/02/14 Permalink
Predict
Melbourne, Victoria
6 posts
Thanks mate for getting back to me very quickly, i appreciate it. Sorry if sounded unsavory, but the issue is just getting to me. Well from my 5 minute average, it says it happened at around 10:50 or so, im not sure if its just a time difference between to two. However I will like to try and resolve why players are getting 'dumped' on a regular basis. So far, this is currently happening to me almost every single day. and can sometimes happen multiple times per day. both servers seem to have this issue.

One thing i do notice is when the server dumps players, people can't connect to the server for 10-15 minutes. It simple throws an error message "Client closed the connection", the exact same issue described in another persons post http://www.ausgamers.com/forums/gsr/thread.php/3382539. I dont know if this information helps you at all.

I wish i had kept a log of when exactly these dumps have been occurring, I have enabled events logging in procon, so I will provide every single drop occurrence that happens after this, to help better assist you with resolving this issue for me and for your other customers.

Thanks in advance.
06:30pm 10/02/14 Permalink
Predict
Melbourne, Victoria
7 posts
Got both my servers populated today, both experienced player drops;

China Rising
11-2-14
Populated @ 3:30PM
Disconnects @ 7:18PM

13-2-14
Populated @ 6:30PM
Disconnects @ 6:49PM

13-2-14
Populated @ 7:00PM
Disconnects @ 7:40PM

13-2-14
Populated @ 8:10PM
Disconnects @ 8:59PM

Popular Maps
11-2-14
Populated @ 7pm
Disconnects @ 8:35:58pm

12-2-14
Populated @ 6:30pm
Disconnects @ 8:49pm
10:10pm 11/02/14 Permalink
dru80c
Newcastle, New South Wales
42 posts
its becoming a big joke why dosent gameservers have this problem?
09:14am 13/02/14 Permalink
BiGGGGRED
Sunshine Coast, Queensland
30 posts
They do have that problem dru80c every provider in aus has been having this problem since day 1 of release . Its a f*****g joke that EA can rent these servers to isp in the state they released the game and expect us customers to pay for them
12:08pm 13/02/14 Permalink
Malthius
Brisbane, Queensland
1203 posts
Got a positive response back from DICE ("can see the problem, can help you find a solution"), so we'll keep working with them.
04:10pm 13/02/14 Permalink
Predict
Melbourne, Victoria
8 posts
I hope they fix this real soon, trying to start my server, had 20 people 15 mins later and the server crashes, absolute joke.
06:50pm 13/02/14 Permalink
LCARSx64
Sydney, New South Wales
21 posts
According to Battlelog Forums, there's going to be both a PC client and server update today starting at 1AM PDT / 9AM UTC, so that should be about 8PM AEST. Hopefully this update fixes the issues, although I wouldn't be surprised if they break something else. :/
07:27pm 13/02/14 Permalink
Predict
Melbourne, Victoria
9 posts
According to Battlelog Forums, there's going to be both a PC client and server update today starting at 1AM PDT / 9AM UTC, so that should be about 8PM AEST. Hopefully this update fixes the issues, although I wouldn't be surprised if they break something else. :/


Yeah can't wait, will post back findings.
07:42pm 13/02/14 Permalink
BiGGGGRED
Sunshine Coast, Queensland
31 posts
lets see if the add Air superiority back into the search funtions
08:32pm 13/02/14 Permalink
Predict
Melbourne, Victoria
10 posts
No luck even with new patch. Ive had 3 player disconnects in a period of 3 friggen hours. This is terrible. Why do other servers not have this problem?
09:06pm 13/02/14 Permalink
dru80c
Newcastle, New South Wales
43 posts
No luck even with new patch. Ive had 3 player disconnects in a period of 3 friggen hours. This is terrible. Why do other servers not have this problem?


try running 10 servers mate, 10 servers flushing all players and "client closed down connection " until multiple restarts get them working again *sigh* then you will know frustration

sometimes they require 8-10 restarts to accept players again, flush players at least 1-2 times an hour
09:26pm 13/02/14 Permalink
Dremark
Sydney, New South Wales
48 posts
No luck even with new patch. Ive had 3 player disconnects in a period of 3 friggen hours. This is terrible. Why do other servers not have this problem?


We just had our 4th disconnect since 730pm (syd).
10:28pm 13/02/14 Permalink
Malthius
Brisbane, Queensland
1204 posts
Spent a lot of time today refining the tools we use to detect these drops, and are working with some very smart networking people to see what we can do while we wait for DICE to get back to us.
04:12pm 14/02/14 Permalink
Predict
Melbourne, Victoria
11 posts
Good to hear progress is being made. Second assault is arriving soon, I hope its solved before then.
06:21pm 14/02/14 Permalink
dru80c
Newcastle, New South Wales
44 posts
across 10 servers, 32 crashes in the last 2 hours

you can check our logs for restarts, just pointing out it is getting a lot worse
06:26pm 14/02/14 Permalink
Predict
Melbourne, Victoria
12 posts
across 10 servers, 32 crashes in the last 2 hours

you can check our logs for restarts, just pointing out it is getting a lot worse


i agree, i populated a server today at 6:30 and its crashed within 45 minutes.
07:13pm 14/02/14 Permalink
dru80c
Newcastle, New South Wales
45 posts
theres obviously something wrong on ausgamers end because my multiplay server has been up for 6 days with no problems.
11:11am 15/02/14 Permalink
Predict
Melbourne, Victoria
13 posts
theres obviously something wrong on ausgamers end because my multiplay server has been up for 6 days with no problems.


Well today it seemed a bit better, one of my servers populated at 11am, at 7:16pm it crashed. Much better but this issue still needs to be badly addressed.

Update. My second server just crashed shortly after my first @ 7:18pm. I dont know if this will help at all.
07:16pm 15/02/14 Permalink
dru80c
Newcastle, New South Wales
46 posts
I cant believe we get charged a whole month rent when we cant even use the servers.
02:11pm 16/02/14 Permalink
Predict
Melbourne, Victoria
14 posts
I cant believe we get charged a whole month rent when we cant even use the servers.


Mate i was just thinking the exact same thing this morning. Populating a server takes time, and if it only stays up for 2 hours then crashes. You basically have to start all over again to populate it. I think as a courtesy, they should give the next month at least half change or even better, one month off.

Servers from other GSPs, from what I have noticed haven't had this problem at all.
02:16pm 16/02/14 Permalink
dru80c
Newcastle, New South Wales
47 posts
Now my ausgamers teamspeak has dropped 3 times today it must be battlefields fault.
06:35pm 16/02/14 Permalink
Oz
Adelaide, South Australia
1 posts
Now my ausgamers teamspeak has dropped 3 times today it must be battlefields fault.


Same here. TeamSpeak gets all distorted then gone, server goes at the exactly same time.
10:12am 17/02/14 Permalink
Malthius
Brisbane, Queensland
1206 posts
There were some giant DDOS over the weekend (not just against us, on the internet generally). We are going through our records to see what parts of it we can "fix" and what parts of it were other ISP's having issues that affected their customers playing on our servers.
11:15am 17/02/14 Permalink
dru80c
Newcastle, New South Wales
50 posts
There were some giant DDOS over the weekend (not just against us, on the internet generally). We are going through our records to see what parts of it we can "fix" and what parts of it were other ISP's having issues that affected their customers playing on our servers.



ok, that explains TS

also our Battlefield servers still dumping players and requiring restarts after *client closed down connection* tried populating this morning but still doing it within half an hour
we tried restarting them everytime but 8 servers requiring constant restarting every hour wore a bit thin after a while

do we have an ETA on a fix ?! we are running quite a large gaming community and its making it very hard with servers that wont accept players or stay full
11:41am 17/02/14 Permalink
Malthius
Brisbane, Queensland
1207 posts
Much of the problem is that the DDOS are so widespread at the moment, including attacks on critical infrastructure in Europe and flooding of links in Australia. This makes it really hard to get the attention of anyone outside our datacentre on our problem - everyone is yelling at DICE, at EA, at various ISPs and transit providers - it is very difficult to pin down where we can make a difference and where we have to cry at our upstream providers.

We are doing everything we can though, including some actual face to face meetings this week.
04:45pm 17/02/14 Permalink
dru80c
Newcastle, New South Wales
51 posts
Much of the problem is that the DDOS are so widespread at the moment, including attacks on critical infrastructure in Europe and flooding of links in Australia. This makes it really hard to get the attention of anyone outside our datacentre on our problem - everyone is yelling at DICE, at EA, at various ISPs and transit providers - it is very difficult to pin down where we can make a difference and where we have to cry at our upstream providers.

We are doing everything we can though, including some actual face to face meetings this week.



Ok, the issue has been happening for weeks though. Just confirming we're to believe it has been DDOS the whole time ?

appreciate you doing everything you can to rectify the situation though
04:47pm 17/02/14 Permalink
Malthius
Brisbane, Queensland
1208 posts
I don't want to say it has been DDOS against AusGamers GSR the whole time, because I don't have the data to confirm or deny - the drops / disconnects I've seen in the past haven't lined up properly with a DDOS on our network.

To be clear, we've had times where there was a DDOS and we had no effect on player numbers at all, and we've had times where there were drops and there was no DDOS against us.

What I can't speak to is whether anyone between us and Blaze (the back end) was DDOSed or otherwise degraded.

What I can say is that over the last week the DDOS and player dropouts / problems have lined up with DDOS attacks either against our network or against Blaze.
04:59pm 17/02/14 Permalink
Predict
Melbourne, Victoria
15 posts
Server today lasted just under 2 hours, every player had increased pings, and then the server completely restarted.
07:52pm 17/02/14 Permalink
PEACE
Christchurch, New Zealand
7 posts
Server today lasted just under 2 hours, every player had increased pings, and then the server completely restarted.

Ours has been doing the same thing high ping then it disconnects and restarts , its not a good look, ;( PEACE
10:43pm 17/02/14 Permalink
Phenom600
Perth, Western Australia
46 posts
Trying hard to keep my mouth shut!, it's something that I believe has been going on for years now, something Brendan and I disagree on ;)

Either way I hope it gets sorted and some stability brought back to the service, as I'd rather be spending my money locally than in the USA.
12:16am 18/02/14 Permalink
dru80c
Newcastle, New South Wales
52 posts
I tried to keep my mouth shut.Over the xmas dramas with servers didnt say nothing just hoped it will be fixed but enough is enough.
10:11am 18/02/14 Permalink
Dremark
Sydney, New South Wales
49 posts
So i guess we are experiencing both the attacks on "backend" and Ausgamers atm?

The "backend" disconnect,where you see everybody get booted from the server(visable in rcon)with error "Game disconnected,client closed down connection" ,server will still be visable in BL yet will have the score board showing "Score board is too busy right now" ,while nobody can rejoin. Server will require a restart to enable players to join.

The Ausgamers disconnect,where rcon disconnects,game server disconnects with players error msg "Game disconnected,something went wrong" ,server will not be visable in BL and will require a restart.

The backend/blaze disconnects have been constant each day for months and seems the Ausgamers disconnects have been occurring for the last couple days.

Just a guess.

We have had 1 of each so far this morning.

Cheers.
10:49am 18/02/14 Permalink
Ner0
Sunshine Coast, Queensland
2 posts
Hi all, glad to be following this post.

I am (obviously) having all the same issues as above. I rent 2 servers for my Gaming Community and quite frankly, it's starting to become a ridiculous situation. I have done everything on my end, (as much as restarting and monitoring server logs) but as of viewing this thread, seems I don't have to look any further...

Weather it is EA/DICE or a ISP. I've been paying for a service, which doesn't serve me. It's rather sad that all of my donations and some of my own coin is going towards something we cannot use. I cannot keep my server populated for roughly 2 hours. I've been experiencing the "Player Dumping" issue. To be honest, I've down right through with the battlefield series after all of the issues through BF3/BF4 server hosting wise. I hope that one-day they can get their crap together. It needs to be done, or they're going to receive a big wake-up call when they lose half of their community to lack of server/game stability. In saying all of that, I highly agree with dru80c & Predict.


Regards,
~iNer0
East Coast Gaming
09:05pm 18/02/14 Permalink
Predict
Melbourne, Victoria
17 posts
Second Assault is on, and my servers are pumping. All i need to do is wait 2 hours and they will all be gone. It's so hard to improve your community when this keep happening to your servers.

Are the affected customers going to be reimbursed?
10:10pm 18/02/14 Permalink
Predict
Melbourne, Victoria
18 posts
Okay one of my Second Assault servers just lasted 30 minutes, this is getting worse guys. Time to look for another GSP i guess. Can't be loyal while this is happening.

Update my second server crashed just minutes after the other one. *Sigh*
10:25pm 18/02/14 Permalink
Dremark
Sydney, New South Wales
50 posts
Okay one of my Second Assault servers just lasted 30 minutes, this is getting worse guys. Time to look for another GSP i guess. Can't be loyal while this is happening.

Update my second server crashed just minutes after the other one. *Sigh*


We just had our 4th crash since Second assualt. Can't get through 1 map.
10:50pm 18/02/14 Permalink
Ner0
Sunshine Coast, Queensland
3 posts
My second server running 24/7 Metro crashed (DUMPED) about 25 minutes ago.

Now, @ 9:48pm My local time - Both of my servers just dumped every player. My 24/7 Conquest which was 32/32+Queue & my Metro Server.

I'm sick to death of this crap. I'm considering to cancel my services with AusGamers and go with Phenom600's (Leader of FPSG Clan) game servers.
10:52pm 18/02/14 Permalink
MrMickeyG
Melbourne, Victoria
34 posts
massive issues tonight, my server had a ping of 124 when it dumped everyone. Clearly this is DDOS. Ausgamers - how can you mitigate this?

I have servers with Hypernia, and NFO (US based). Hypernia was also dumping players like a MOFO, clearly DDOS. NFO hasnt crashed, but laggy-ness can be an issue.

Brendan - we need help....
10:53pm 18/02/14 Permalink
Predict
Melbourne, Victoria
19 posts
I'd love to get my hands on the people responsible for the DDoS. If only it were easy.
11:00pm 18/02/14 Permalink
LCARSx64
Sydney, New South Wales
24 posts
I'd love to get my hands on the people responsible for the DDoS. If only it were easy.

Yeah it would be awesome to be able to ban them from ever using the internet again ... oh and to make them pay for the downtime of everyone's servers. ;)
11:13pm 18/02/14 Permalink
Malthius
Brisbane, Queensland
1211 posts
We have some changes in progress to respond to the current batch of DDOS attacks that we expect will restore a great deal of stability on our end, but we are aware of continued attacks against Blaze as well, which are beyond our control. I'm expecting an update on this this afternoon or tomorrow morning.
02:30pm 19/02/14 Permalink
Phenom600
Perth, Western Australia
47 posts
I wouldn't jump ship and do what we are doing, for us it was our last resort to keep our clan going as this has been happening to us for about 3x years now, and it's only ever let up for a few months at a time.... I was at the point of giving up.

Our USA based instances have gone down 4x times today from blaze disconnects, so it's by no means a perfect solution. It's probably a minimum of twice a day that this happens, sometimes it's within 30-40minutes of seeding the server again and it's totally random.

Can't say I've heard the rumours that the blaze back end is being DDoSed from any other GSP, I do have a message that states about 30% of server connections were dropped at a specific time today but that's about all, and it's not uncommon for this to occur.

Either way, BF server IPs are whitelisted, so it's not exactly hard to put filters in place to restrict connectivity only to those hosts, there's a few ways of doing this and all generally have minimal performance impact (boarder router)..

ICMP is often rate limited or set to a lower priority so a high ping doesn't necessarily relate to an attack. We had to disable our latency manager over all of our instances as users were constantly being kicked for high pings, users that generally had a ping of 5-10ms were spiking to 500ms+ at times and this wasn't unique to a specific ISP, this was when our servers were working.

Most of the attacks against our kit in the USA (forums/teamspeak) and our battlefield servers where I have access to view what's going on, or read the reports. Almost all of those attacks are NTP reflection attacks, with the occasional vulnerability attack (level 7 attacks). I would say 98-99% of these attacks are only 1-3Gbit at best, with the odd huge one coming through that can cause issues, so 99% of the time the attacks are mitigated and have no impact on our players.

(Disclaimer I don't work for ausgamers so this is speculation)

Granted I can't see someone hitting Ausgamers with 10-70Gbit DDoS attacks all day long, attacks of this size aren't as common as people might think, and if they were that large they'd be causing much bigger problems and you'd likely see IPs being null-routed , I've come to the conclusion that there's absolutely no mitigation equipment in place at all for the game servers.

Or the people configuring it have nfi what they are doing, either way we are never likely to find out.
06:38pm 19/02/14 Permalink
Malthius
Brisbane, Queensland
1212 posts
Granted I can't see someone hitting Ausgamers with 10-70Gbit DDoS attacks all day long, attacks of this size aren't as common as people might think, and if they were that large they'd be causing much bigger problems and you'd likely see IPs being null-routed , I've come to the conclusion that there's absolutely no mitigation equipment in place at all for the game servers.

Or the people configuring it have nfi what they are doing, either way we are never likely to find out.


Aww, your post was going so well, then you had to ruin it by being both obviously wrong and needlessly offensive.
06:46pm 19/02/14 Permalink
Phenom600
Perth, Western Australia
48 posts
Post up the graphs, it would go totally against what I've seen over the past fifteen years of my career of which two years was spent at Telstra, it also goes against what most security experts publish in terms of the average attack strength.

What equipment do you guys use to mitigate attacks? telling me if it's Arbor, NSFOCUS or X provider shouldn't be a problem.... I just get told it's all a big secret hence the speculation and doubt that anything is in place.

I've shown you mine before ;) I've also gone into detail about the equipment in place, in fact most of the time we post the information on the forum so our player base has an idea as to what's going on.

I've tried every GSP in AU and fed that information back to you with regards to performance and how we were attacked constantly, yet there seems to have been a fair bit of denial that it was happening here.

Either way, I'm sure I won't get any of the above questions answered, I just hope the issue gets resolved so I can come back and lodge more tickets just to annoy you!.
07:38pm 19/02/14 Permalink
Malthius
Brisbane, Queensland
1213 posts
I can't talk about the hardware because it isn't our hardware, and frankly, providing that sort of information would be a commercial advantage to our opposition. I've spent the last 18 months being as open with you as I can, and I thought that I'd earned enough credibility that you wouldn't simply call me a liar.

Saying that I am lying and that there is no DDOS mitigation in place because I won't talk about specifics is... disappointing. I'm sure you are capable of running constant traces to our servers and seeing the routing change when the mitigation gear kicks in; in fact, I'd be surprised if you haven't already done that.

I will ask if I can get some numbers or graphs to share, but frankly I doubt I'll be permitted, there are too many layers of NDAs and commercial confidentiality agreements.
12:32pm 20/02/14 Permalink
Predict
Melbourne, Victoria
20 posts
We have some changes in progress to respond to the current batch of DDOS attacks that we expect will restore a great deal of stability on our end, but we are aware of continued attacks against Blaze as well, which are beyond our control. I'm expecting an update on this this afternoon or tomorrow morning.


Hopefully on what im about to say won't jynx me or the rest of us. Servers only crashed once yesterday, and so far today from 11am, my servers haven't crashed at all. Keep up the good work Malthius and AusGamers!
05:59pm 20/02/14 Permalink
Phenom600
Perth, Western Australia
49 posts
Brendan, I wasn't calling you a liar, at least that wasn't my intention... I understand what the situation is with regards to Ausgamers and Telstra and you've explained the level of access you guys have to the equipment in the past, so unless things have changed that's rather limited which probably makes it harder on you.

If Telstra is using BGP to redirect traffic to scrubbing appliances in the event of an attack it's not an appropriate solution for "gaming servers" especially of the bf4 nature granted you only need to look at them sideways and all 64 players drop off.

I've run traceroutes in the past and haven't noticed any changes, however I was really only interested in the increase in latency and where it was occurring (last visible hop on your end) so would have missed it.

With regards to a commercial advantage in AU I don't think you guys have anything to worry about, there's only one provider who has DDoS mitigation gear in place and it's actually done through the USA (blacklotus) hence why all traffic is filtered through that path first, or at least that's what they tried, of course it's not the easiest solution to implement due to the nature of the Internet.
06:49pm 20/02/14 Permalink
Dremark
Sydney, New South Wales
51 posts


Hopefully on what im about to say won't jynx me or the rest of us. Servers only crashed once yesterday, and so far today from 11am, my servers haven't crashed at all. Keep up the good work Malthius and AusGamers!


It's all your fault :)

We had a good run for a couple days,only a few disconnects.

Back to normal now.
09:36am 21/02/14 Permalink
Predict
Melbourne, Victoria
21 posts
Yeah haha, mine crashed about an hour after i said that. But having said that, no crashes today so far, great work AusGamers!
06:06pm 21/02/14 Permalink
Predict
Melbourne, Victoria
22 posts
One of my servers experienced 3 crashes today, looks like the problem is coming back
07:21pm 22/02/14 Permalink
Malthius
Brisbane, Queensland
1218 posts
One of my servers experienced 3 crashes today, looks like the problem is coming back


Not our fault, DICE has had some back end issues with Blaze.

This is a huge part of the problem - from your point of view the server goes down, but there are so many links in the chain that can fail to cause that disconnection.
11:42am 24/02/14 Permalink
Malthius
Brisbane, Queensland
1219 posts
Brendan, I wasn't calling you a liar, at least that wasn't my intention... I understand what the situation is with regards to Ausgamers and Telstra and you've explained the level of access you guys have to the equipment in the past, so unless things have changed that's rather limited which probably makes it harder on you.

If Telstra is using BGP to redirect traffic to scrubbing appliances in the event of an attack it's not an appropriate solution for "gaming servers" especially of the bf4 nature granted you only need to look at them sideways and all 64 players drop off.


Well, when I say "DDOS stuff is being tweaked" and you say "there isn't any DDOS stuff", there is a pretty clear inference...

I can't confirm or deny what the hardware does, but what I will say is that it worked in the past - we were able to entirely prevent DNS reflection attacks from having any effect on server stability, so whatever we do have in place was entirely appropriate. It took a little time to get it tweaked to the point where it was protecting the servers - if we dial the knob too far to the left it won't protect the servers from all the attacks, and too far to the right and it sees legitimate traffic as an attack and disconnect the servers itself. (Funny example - I was staging the files for a patch deployment and triggered an alert due to traffic size, stopping me from staging the package).

We are now going through the same process for the NTP reflection attacks. Same as everyone else in the known universe.
11:50am 24/02/14 Permalink
Phenom600
Perth, Western Australia
50 posts
You're not tweaking anything though right? so how do you know it is actually being done, that was more of the point I was trying to make.

Glad it works, we are getting hit with ~100Gbit attacks now in the USA which is causing packet loss for the entire DC ;) so perhaps the best time to come back to let you deal with it.
07:08pm 24/02/14 Permalink
Malthius
Brisbane, Queensland
1220 posts
I don't get to play with the big metal toys, no, but it is inconceivable that they are telling me that specific changes are being made while not actually doing anything. I give them too much hard data on player counts (which are polled every 10 seconds) and disconnections to try to do some sort of placebo thing.
04:58pm 26/02/14 Permalink
Dremark
Sydney, New South Wales
55 posts
Update:

Seemed to be doing alot better over the last week,only a couple disconnects each day and the the odd Query Failure.

Most disconnects happen during the morning till around midday and from 22:30pm through till midnight.

Almost without fail we will have one each night at 23:50/Midnight.

Cheers.
12:59pm 03/03/14 Permalink
Dremark
Sydney, New South Wales
56 posts
Almost without fail we will have one each night at 23:50/Midnight.


Anyone else having the same issue?
11:57pm 03/03/14 Permalink
taco-f-kisses
1 posts
surely Mammoth Media provide a layer of security for their infrastructure & DCs?? and SLAs are in place? and if it is as you say an EA/DICE backend issue then surely any servers connected to that backend BLAZE node would ALL go down resulting in a mass exodus of servers that geographically similar?

I'm not looking to start an argument, but I'm sure if BigPond Movies started crapping out as much as this then both Mammoth Media & Telstra would be on the case, there would be an investigation & if need be new infrastructure deployed to combat the issue.


and as for


I can't confirm or deny what the hardware does, but what I will say is that it worked in the past - we were able to entirely prevent DNS reflection attacks from having any effect on server stability, so whatever we do have in place was entirely appropriate. It took a little time to get it tweaked to the point where it was protecting the servers - if we dial the knob too far to the left it won't protect the servers from all the attacks, and too far to the right and it sees legitimate traffic as an attack and disconnect the servers itself. (Funny example - I was staging the files for a patch deployment and triggered an alert due to traffic size, stopping me from staging the package).


that is normal with any IT security solution and is why most organisations deploy multiple layers and different technologies to overcome this.

Maybe it is time to deploy some smarter smarts :)


I would also just like to add, that IF there are any SLAs in place, even a very relaxed 99.95% that is ONLY 5.04 minutes of downtime per week..., this SERIOUSLY requires some attention, not only to mention that this is a PAID service!!!
12:15am 05/03/14 Permalink
dru80c
Newcastle, New South Wales
55 posts
ok we are up to 6 crashes on 4/8 of our servers already today and it is only 9am

i am just about over this completely! we have 8 servers with you guys and 3 with other providers (which do not crash nearly as regularly )
remind me, why are putting up with this and have been for 6 months ?
09:10am 05/03/14 Permalink
Predict
Melbourne, Victoria
23 posts
Server crashes only after 3-4 hours after populating. This is terrible.
07:42pm 06/03/14 Permalink
Phenom600
Perth, Western Australia
51 posts
Last time I posted about this all my instances started going offline non stop... but here goes, my Ausgamers instance has only been offline once I believe in the last few days, mind you it's not really getting a huge workout either like it used to.

The new maps if you have them in your rotation are buggy as all hell and are causing server crashes, we were seeing 12+ a day on Caspian, Oman crashes a few times a day as well.

R28 update now so hopefully that makes things a little more stable, sorry couldn't help but laugh ;(
08:35pm 06/03/14 Permalink
taco-f-kisses
2 posts
is Adam Williams aware of how bad of a reputation AUSGAMERS is getting due to this issue?

If not let me know & I'll be more than happy to pick up the phone to him & discuss it
08:37pm 06/03/14 Permalink
Phenom600
Perth, Western Australia
52 posts
If you're going to waste your time calling someone at least make it worth your while and call Michelle...
08:33pm 07/03/14 Permalink
Phenom600
Perth, Western Australia
53 posts
Ok Brendan,

Our Ausgamers instance went down, most definitely a DDoS attack as I got alerts through about our other 5x instances being hit at exactly the same time.

Unfortunately a small amount of traffic got through and dropped our lockers instance (USA), by small I'm talking not much at all and only for a split second. I was surprised at how little traffic it took to drop the instance (bandwidth graphs), DICE need to do something about that part.

Flood types were TCP RST/SSYN time was ~12:03 EST
12:50am 08/03/14 Permalink
Phenom600
Perth, Western Australia
54 posts
Keep getting this (and everyone else), even after a restart....

Error Game disconnected: Client closed down connection. Check your network connection and make sure that port 3659 (UDP) is open.
01:33pm 08/03/14 Permalink
Predict
Melbourne, Victoria
24 posts
Yeap, I am getting that issues multiple times per day, per server. So this issue is also occurring on non-Ausgamers servers?
09:15pm 08/03/14 Permalink
Phenom600
Perth, Western Australia
55 posts
Meh I gave up seeding mine, got it going a dozen or so times and it was taken offline each time, then couldn't even join the instance.

Same s*** that's been happening for a long arse time to us.

I pretty much have the "packetloss icon" on my screen non stop, same with other players, checking it out last night there was ~2% packetloss occurring on the server for an hour or so that I tested.

Unfortunately I can't check the path back....
11:09pm 08/03/14 Permalink
Phenom600
Perth, Western Australia
56 posts
Brendan,

I've turned our instance off as it's becoming extremely embarrassing keeping it going.

It's been seeded a lot this weekend, each and every time it gets going it's taken offline like clockwork.

Let me know when if and when it can be fixed, otherwise buy yourself a beer with the $70 bucks....
01:47pm 09/03/14 Permalink
Predict
Melbourne, Victoria
25 posts
Malthius. Are you hounding DICE for a fix? Im disconnecting like this going out of fashion.
04:22pm 09/03/14 Permalink
MrMickeyG
Melbourne, Victoria
35 posts
ours do the same, I cant be bothered anymore, TitanFall is preloading right now!
05:16pm 09/03/14 Permalink
Green
Sydney, New South Wales
1 posts
I have tried to be patient for months, waiting for a fix. I acknowledge that Ausgamers are advising the problem rests with Dice however I can't keep a server populated for more than a couple hours lately and it is then black listed for days. Good luck to Ausgamers - I might be back if the stability is sorted in the future. For now, I'm done. Cancelled.
07:36pm 10/03/14 Permalink
Phenom600
Perth, Western Australia
57 posts
I sent this through to Malti, but I will also post it here, I generally see this whenever my server goes down.

Out of 350 packets sent, only 34 were received by the server, I've got over 1000+ packets sent right now and we are up to about 80 packets received at the host.

All the packets from my ISP are hitting telstras network just fine, it's from then on things go south, specifically at the bf4 host.


|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| fritz.box - 0 | 350 | 350 | 0 | 0 | 5 | 0 |
| 074.pth0304.pth.iprimus.net.au - 1 | 346 | 345 | 6 | 21 | 70 | 24 |
| 193.pth0306.pth.iprimus.net.au - 1 | 342 | 340 | 7 | 24 | 78 | 24 |
| ge-2.br01.pth.iprimus.net.au - 1 | 346 | 345 | 7 | 23 | 81 | 28 |
| ge0-1.br01.pth.iprimus.net.au - 1 | 346 | 345 | 7 | 21 | 118 | 8 |
| 119.225.139.201 - 1 | 342 | 340 | 8 | 27 | 141 | 23 |
| No response from host - 100 | 70 | 0 | 0 | 0 | 0 | 0 |
| 59.154.142.6 - 1 | 338 | 335 | 8 | 21 | 101 | 14 |
| Bundle-Ether11.wel36.perth.telstra.net - 1 | 346 | 345 | 8 | 24 | 73 | 29 |
|bundle-ether4.wel-core3.perth.telstra.net - 1 | 346 | 345 | 9 | 24 | 70 | 15 |
|bundle-ether7.fli-core1.adelaide.telstra.net - 1 | 342 | 340 | 41 | 56 | 99 | 43 |
|bundle-ether9.win-core1.melbourne.telstra.net - 2 | 334 | 330 | 51 | 64 | 111 | 65 |
|bundle-ether12.ken-core4.sydney.telstra.net - 3 | 322 | 315 | 64 | 77 | 119 | 69 |
| No response from host - 100 | 70 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 70 | 0 | 0 | 0 | 0 | 0 |
| ga0114.gamearena.com.au - 65 | 97 | 34 | 0 | 129 | 169 | 161 |
|________________________________________________|______|______|______|______|______|______|
WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
05:00pm 16/03/14 Permalink
taco-f-kisses
3 posts
11:06pm 18/03/14 Permalink
Phenom600
Perth, Western Australia
58 posts
I've used many of F5's product offerings over the years and I would never recommend them for DDoS mitigation. If you're going to recommend a vendor at least mention Arbor or NSFOCUS, specifically Arbor products which work very well.

Having said that I was told it wasn't an attack, and it's pretty hard to argue otherwise when you have no access to logs or bandwidth graphs ;) but yes all the symptoms are there, as they have been many times in the past.

On kit we do have access to log files, including interface dumps and bandwidth graphs. I can confirm that attacks on those instances correlate to outages on our Ausgamers instances (now instance) in the past.

This has also been confirmed by 3x other RSP's within AU when pushed on the issue, however Ausgamers are saying otherwise when I question the outages, in fact I would say Brendan is probably sick of responding to me by now ;)
08:01am 19/03/14 Permalink
dru80c
Newcastle, New South Wales
61 posts
It beyond a joke we shouldnt pay full price for something we do not have.Its at the point where a ombudsman should be contacted if we get charged full price.
04:23pm 20/03/14 Permalink
Predict
Melbourne, Victoria
26 posts
Server today lasted 1 hour, then ultimately boots everyone off the sever, now i can't populate it anymore. Cancelling looks more and more appealing everyday.
06:35pm 20/03/14 Permalink
dru80c
Newcastle, New South Wales
62 posts
there f***ed i have 7 and they have stuffed up like 8 times today all of them.
07:38pm 20/03/14 Permalink
Predict
Melbourne, Victoria
27 posts
Hmm, the same server i managed to get it populated again, it lasted 30 mins. Wow.
07:39pm 20/03/14 Permalink
Dremark
Sydney, New South Wales
61 posts
10:55pm 20/03/14 Permalink
Dremark
Sydney, New South Wales
62 posts
Edit:Double post.
10:57pm 20/03/14 Permalink
Phenom600
Perth, Western Australia
59 posts
My gameservers instance is actually worse, I manage 1-2 rounds before it goes arse up and then wait ~40+ hours hours for some generic response and it costs $30 a month more than here.

The strange thing is no "lost connections" in the console or nothing, just 64 players disappearing into oblivion. At least on our Ausgamers instance we see all the players losing their connection and it's pretty much expected.

Off topic a bit but does everyone else have crap loads of rubber banding in-game?
04:32pm 21/03/14 Permalink
Dremark
Sydney, New South Wales
63 posts
My gameservers instance is actually worse, I manage 1-2 rounds before it goes arse up and then wait ~40+ hours hours for some generic response and it costs $30 a month more than here.

The strange thing is no "lost connections" in the console or nothing, just 64 players disappearing into oblivion. At least on our Ausgamers instance we see all the players losing their connection and it's pretty much expected.

Off topic a bit but does everyone else have crap loads of rubber banding in-game?


No rubber banding at all for us.

Running Dragon,Paracel and SA maps in rotation also,oh... but.. NO METRO! ;)
04:46pm 21/03/14 Permalink
Phenom600
Perth, Western Australia
60 posts
I couldn't run Dragon here, US dedi no problem at all but not here 100% CPU on the graph and rubber banding galore.

SA maps here and our instance crashed all the time so we stopped it, tried it on GS twice and they dropped all players, US dedi didn't like Caspian but the rest worked fine so it's odd that there's so many different experiences when it's all of the same image.

Metro really isn't that popular any more, nothing like it used to be that's for sure.
05:03pm 21/03/14 Permalink
Dremark
Sydney, New South Wales
64 posts
I couldn't run Dragon here, US dedi no problem at all but not here 100% CPU on the graph and rubber banding galore.

SA maps here and our instance crashed all the time so we stopped it, tried it on GS twice and they dropped all players, US dedi didn't like Caspian but the rest worked fine so it's odd that there's so many different experiences when it's all of the same image.

Metro really isn't that popular any more, nothing like it used to be that's for sure.


We did have the high CPU lag after R28 for a week.

We disconnect on every map,no real pattern to it,start of round,mid round,end round,map change,5 players or 64..really random.

Longest we have been running without a disconnect is around 24h..and that was last weekend. Atm we average 6 disconnects a day.

Metro will always be popular..i think its more BF4 as a whole that is slowly dying...no surprise really.
11:11am 22/03/14 Permalink
Dremark
Sydney, New South Wales
65 posts
Ok,

We now have our server disconnecting at 11 minutes past the hour every hour.

Started Wednesday 26th and was every 3 hours without fail.

Last night (friday 28) we had our normal 19:11 disconnect then 21:11,22:11,23:11.. etc..and been every hour since.

Anyone else having same issue?
01:23pm 29/03/14 Permalink
dru80c
Newcastle, New South Wales
64 posts
yes yes 6 servers all stuffed some you cant use but aus gamers send me invoices when they dont even reply to my tickets.
02:07pm 29/03/14 Permalink
Static
17 posts
Main server dropped all players x 3 today, 3 servers un contactable via web "Server 500 error" (hosted on different hosts) at 19:17 this evening.
203.46.105.22
203.46.105.24
203.46.105.72

Fix your network please, find where it broken and fix it. It aint rocket science.

There are many free network monitoring tools available that can pin point where the issue is. If it is DDOS please let us know don't hid it.
07:26pm 30/03/14 Permalink
Static
18 posts
And again. FFS. 3 servers not in Battlelog. GameServers and Games.on.net servers are still full
08:17pm 30/03/14 Permalink
Dremark
Sydney, New South Wales
66 posts
Main server dropped all players x 3 today, 3 servers un contactable via web "Server 500 error" (hosted on different hosts) at 19:17 this evening.
203.46.105.22
203.46.105.24
203.46.105.72

Fix your network please, find where it broken and fix it. It aint rocket science.

There are many free network monitoring tools available that can pin point where the issue is. If it is DDOS please let us know don't hid it.


Seems all of Ausgamers went down just now.

Back up for a few minutes then down again.

We have had 20 today so far...every hour to the minute...

Edit: 24 disconnects.
08:20pm 30/03/14 Permalink
Static
19 posts
YOU HAVE NO FAVORITE SERVERS
Add servers to your favorites by clicking the star in the list.

Strange i had them a few minutes ago....


Oh wait they are all AusGamer servers.

Cant even get to the GameCreate Interface.
08:31pm 30/03/14 Permalink
LCARSx64
Sydney, New South Wales
28 posts
YOU HAVE NO FAVORITE SERVERS
Add servers to your favorites by clicking the star in the list.

Strange i had them a few minutes ago....


Oh wait they are all AusGamer servers.

Cant even get to the GameCreate Interface.

Yep same, I can't get to GameCreate either. :(
08:38pm 30/03/14 Permalink
Static
20 posts
Rank Server Name Players IP:Port
2854. [CYN8] Operation Bottleneck HC (BigPond Unmetered) Down 203.46.105.72:22000
404. [CYN8] Armour Air Maps HC FastV #1 (BigPond Unmetered) Down 203.46.105.22:21270

Please note the DOWN Indication.
09:11pm 30/03/14 Permalink
Static
21 posts
Big Lag spike affecting both 203.46.105.22 and 203.46.105.24 at 18:56

Anything to report AusGamers staff?
07:00pm 31/03/14 Permalink
dru80c
Newcastle, New South Wales
65 posts
lol at ausgamers we dont care about customer satisfaction we just want your money.
12:57pm 01/04/14 Permalink
Predict
Melbourne, Victoria
28 posts
Lol i cant keep a server running for more than 30 mintues with the new DLC. Both servers keep crashing out. Weird since the past few days the server has been pretty good.
07:07pm 01/04/14 Permalink
SnotGoblin
New Zealand
46 posts
Funny stuff reading all this :)

All GSP's having similar issues lads, I have servers with Hypernia, GameServers, I3D.net etc etc for testing purposes.....

All have the same issues as AusGamers, the only real difference is that they have a support system, and you get a response fairly quickly
(Hypernia the best @ responding to tickets by a country mile! AusGamers the worst also by a country mile!)

Had issues on the Hypernia network similar to above where all servers would crash at the same time every hour.
GameServers have been good, but there servers "dont feel" as smooth as the other GSP's, Hit reg is a bit dodgier than normal.
I3D.net same issues as everyone else, but support system is pretty good.

I think Phenom has done well by putting his servers over @ NFO, at least he has the control in his hands and can see what is going on behind the scenes a bit more,
the rest of us are at the mercy of the support systems of various GSP's :)

I actually blame EA/DICE....... not pre-ordering the next BF game.....
07:17am 15/04/14 Permalink
Phenom600
Perth, Western Australia
63 posts
Hey I did tell everyone in one or two of these threads not to jump ship as the other GSPs are the same or worse ;)

It's interesting that you had a relatively positive experience with Hypernia as for us they were most definitely the worst GSP we've ever used in AU and we've used all of them multiple times over.

The reality is that the only GSP in AU that has a large network (owned/operated) behind them is Ausgamers, gameservers uses internodes/iinets and the rest are simply purchasing cheap arse xeon 1230s from dedicatedgaming or servers Australia.

While i3d does have a limited network presence in AU, it has next to no capacity which is why they run their servers off of dedicatedgamings network.

NFO has been an amazing experience, seriously I can't tell you how fantastic these guys are, if I ever did need anything changed they would respond to the ticket within two minutes (24/7) and 99% of the time fixed it.

On the DDoS side of things, when we moved to them our instances were all being attacked daily, I was able to see and capture this information, view it on the network graphs and they generally detected it and alerted me as well. I haven't actually had an attack on our US instances in a while now as the people doing it have given up on those instances and just focus on our AU based ones.

We will however be shutting our NFO dedicated down, $600+ a month on US and AU based instances is simply not sustainable and some of our clan members don't want to play on US based instances so we will be back to being AU based 100%. It's a shame as I was enjoying not having to restart instances all the time or coming online to the 30+ BL messages asking why the instance went down and their scores were lost.
08:25am 15/04/14 Permalink
SnotGoblin
New Zealand
47 posts
Interesting stuff and thanks for sharing :)

Weird as out of all the hosts we have tried, Hypernia have been the best support wise, and their servers feel the best resource allocation wise.
GameServers is a mixed experience, vowed never to use them again, but have 2 servers with them currently, support is slow (5hrs), and the worst thing about GameServers is they have not got it sorted so when you get the blaze disconnect you lose all your stats. Hypernia would always record this as a draw on you battlereport, Gameservers you just lose the stats, which of course pisses people off more!

I feel sorry for all GSP's they have been hung out to dry by EA/DICE and s***** budget server build. This game from a back end perspective is still not ready to actually make a final release retail wise, six months after its release haha!
All GSPs suffering the same symptoms and s***, and suffering the same "What Problem?" from DICE lol.

From an admin perspective my BF4 experience has certainly soured my view of Battlefield and future releases.
Hopefully the original crew from 1942 come together under a different name, and make a proper game!! OR at least the guys who made DC :)

The main thing that bugs me with AusGamers = zero support, you get treated with complete silence, which is shocking from a customer relations perspective.
A "we have no idea what is causing the problem, but we are hassling EA/DICE" even if it is complete BS, is better than absolute silence!!! :)

I like probably every other AU admin wants to support a local GSP, and we dont expect everything to be perfect!
But we do expect answers!! :)

We have decided to cut back to a couple of servers long term, and on different GSP's,
They all have issues, and all suffer DDOS, Its just a matter of spreading the load so you hopefully have a server up for you clannies at some point.
Putting all your eggs in one basket means they all go down if there is a problem!

Cheers,

Snotty
09:24am 15/04/14 Permalink
JWMutant
Adelaide, South Australia
26 posts
Ok so the last 4 days has been horrific, we cant sustain a server more than an hour without a Blaze Disconnect.

Any thoughts if Ausgamers is actually still experiencing ddos, or "working with DICE" on a fix?

At the moment it seems the only option is to cancel the lot, as paying for something we can't use for more than an hour at a time is ludicrous.
12:56pm 21/04/14 Permalink
dru80c
Newcastle, New South Wales
70 posts
Your lucky mine havent worked for over a month i had to stop them all.
08:50am 23/04/14 Permalink
MAG
Other International
40 posts
Has anyone found a host here in OZ that is not having these problems. We are currently using Gameservers and our server is almost unusable and we can't get a bloody reply from support. We are thinking of moving but I'm buggered if I can work out who to move to.
01:19pm 23/04/14 Permalink
SnotGoblin
New Zealand
48 posts
LOL AusGamers are doing nothing for you chaps sadly other than taking your money and laughing IMHO :)

I turned my server off, completely unuseable on the AusGamers network, and Zero support, I asked for a re-install to try to resolve myself, but that never happened.
Big Fat Zero support sadly here, and pretty much the same price as every other GSP, and they actually provide support, even if slow sometimes...

We run 2 servers with GameServers currently, uptime has been pretty damn good. If you have servers with a GSP, and have issues with repeat blaze disconnects,
or UDP port errors, then ask your GSP for a complete re-install. It usually sorts out most of the issues (I reinstalled mine till I was happy with stability).
Most GSP's let you do the re-install process yourself with the click of a button, not AusGamers though, you are screwed, even asking for one doesn't get it.

We have one Server with I3D.net, and thats pretty stable as well, but we purely use that server for clan nights and s**** and giggles.
We purchased a test server with NFO in the states as per FPSG lads, and must say , they make every other GSP look stupid.
Longest time I waited for a response to a ticket @ NFO? 5mins. The shortest time for a response? 1 min.
Problems all solved right there, right then, when you want them solved. Service like you would not believe to be fair... THE BEST GSP WITHOUT FAIL....

We will be turning off our I3D.net server and NFO server this month, as they were purely testing and eval servers.
We will stick with the 2 GS servers, and have decided to branch out into more game types, as our clan dont see a long term future now with the BF series,
the way EA/DICE shafted everyone with this game has put us off. We actively encourage our clannies to not pre order the next game in the series,
its the best way we the community can give the big finger to EA/DICE, and actually get them to listen IMHO.

Cheers.

Snotty
07:47am 28/04/14 Permalink
system
Internet
--
07:47am 28/04/14 Permalink
AusGamers Forums
Show: per page
1
This thread is archived and cannot be replied to.