I've just reviewed the log files for one of your servers for the last week.

I classify events where groups of players are removed from the server into two types, that I've very scientifically named "dumps" and "drops".

A "drop" is where the BF4 server loses connection to the back end ("drops" the back end connection) and reverts into a pre-game mode, disconnecting every player and staying offline for up to 2 minutes after the connection is restored.

Your server had 9 "drops" in the last week, but none of those removed any players, they were all at times when there was nobody online (e.g. 2014-02-08T14:22:38 or 2014-02-09T02:29:31).

A "dump" is where the BF4 server stays online and connected to the back end, but a cluster of players are removed from the server by the back end. (players are "dumped"). The server doesn't report any problem, it just sees a message from the back-end (BattleLog) to remove player #2, then player #3, then player #5 etc, all in within a few seconds.

Your server had 7 "dumps" in the last week, though most of them were earlier in the week and didn't kill the server - e.g. at 2014-02-04T21:58:27 23 of the 57 players were removed.

I'm suspect the one that caused you to make this post was 2014-02-09T11:56:15 where 59 of the 64 players were dumped.

----------

The reason this hasn't been "fixed" is that it is entirely outside our control - it appears to be a network problem affecting the back end servers. DICE are working on the problem and have promised us both a short term improvement and a long term permanent fix, but I don't have any ETA on either of those things. I have sent the log files for another server that is behaving like yours to DICE, and I'll hound them again for an answer later tonight (when they get back to work, it is still Sunday evening over there).

Got both my servers populated today, both experienced player drops;

China Rising
11-2-14
Populated @ 3:30PM
Disconnects @ 7:18PM

13-2-14
Populated @ 6:30PM
Disconnects @ 6:49PM

13-2-14
Populated @ 7:00PM
Disconnects @ 7:40PM

13-2-14
Populated @ 8:10PM
Disconnects @ 8:59PM

Popular Maps
11-2-14
Populated @ 7pm
Disconnects @ 8:35:58pm

12-2-14
Populated @ 6:30pm
Disconnects @ 8:49pm

They do have that problem dru80c every provider in aus has been having this problem since day 1 of release . Its a f*****g joke that EA can rent these servers to isp in the state they released the game and expect us customers to pay for them

I hope they fix this real soon, trying to start my server, had 20 people 15 mins later and the server crashes, absolute joke.

According to Battlelog Forums, there's going to be both a PC client and server update today starting at 1AM PDT / 9AM UTC, so that should be about 8PM AEST. Hopefully this update fixes the issues, although I wouldn't be surprised if they break something else. :/

Yeah can't wait, will post back findings.

No luck even with new patch. Ive had 3 player disconnects in a period of 3 friggen hours. This is terrible. Why do other servers not have this problem?

No luck even with new patch. Ive had 3 player disconnects in a period of 3 friggen hours. This is terrible. Why do other servers not have this problem?

We just had our 4th disconnect since 730pm (syd).

Good to hear progress is being made. Second assault is arriving soon, I hope its solved before then.

across 10 servers, 32 crashes in the last 2 hours

you can check our logs for restarts, just pointing out it is getting a lot worse

i agree, i populated a server today at 6:30 and its crashed within 45 minutes.

theres obviously something wrong on ausgamers end because my multiplay server has been up for 6 days with no problems.

Well today it seemed a bit better, one of my servers populated at 11am, at 7:16pm it crashed. Much better but this issue still needs to be badly addressed.

Update. My second server just crashed shortly after my first @ 7:18pm. I dont know if this will help at all.

I cant believe we get charged a whole month rent when we cant even use the servers.

Mate i was just thinking the exact same thing this morning. Populating a server takes time, and if it only stays up for 2 hours then crashes. You basically have to start all over again to populate it. I think as a courtesy, they should give the next month at least half change or even better, one month off.

Servers from other GSPs, from what I have noticed haven't had this problem at all.

Now my ausgamers teamspeak has dropped 3 times today it must be battlefields fault.

Same here. TeamSpeak gets all distorted then gone, server goes at the exactly same time.

There were some giant DDOS over the weekend (not just against us, on the internet generally). We are going through our records to see what parts of it we can "fix" and what parts of it were other ISP's having issues that affected their customers playing on our servers.

ok, that explains TS

also our Battlefield servers still dumping players and requiring restarts after *client closed down connection* tried populating this morning but still doing it within half an hour
we tried restarting them everytime but 8 servers requiring constant restarting every hour wore a bit thin after a while

do we have an ETA on a fix ?! we are running quite a large gaming community and its making it very hard with servers that wont accept players or stay full

Much of the problem is that the DDOS are so widespread at the moment, including attacks on critical infrastructure in Europe and flooding of links in Australia. This makes it really hard to get the attention of anyone outside our datacentre on our problem - everyone is yelling at DICE, at EA, at various ISPs and transit providers - it is very difficult to pin down where we can make a difference and where we have to cry at our upstream providers.

We are doing everything we can though, including some actual face to face meetings this week.

Ok, the issue has been happening for weeks though. Just confirming we're to believe it has been DDOS the whole time ?

appreciate you doing everything you can to rectify the situation though

Server today lasted just under 2 hours, every player had increased pings, and then the server completely restarted.

Trying hard to keep my mouth shut!, it's something that I believe has been going on for years now, something Brendan and I disagree on ;)

Either way I hope it gets sorted and some stability brought back to the service, as I'd rather be spending my money locally than in the USA.

So i guess we are experiencing both the attacks on "backend" and Ausgamers atm?

The "backend" disconnect,where you see everybody get booted from the server(visable in rcon)with error "Game disconnected,client closed down connection" ,server will still be visable in BL yet will have the score board showing "Score board is too busy right now" ,while nobody can rejoin. Server will require a restart to enable players to join.

The Ausgamers disconnect,where rcon disconnects,game server disconnects with players error msg "Game disconnected,something went wrong" ,server will not be visable in BL and will require a restart.

The backend/blaze disconnects have been constant each day for months and seems the Ausgamers disconnects have been occurring for the last couple days.

Just a guess.

We have had 1 of each so far this morning.

Cheers.

Second Assault is on, and my servers are pumping. All i need to do is wait 2 hours and they will all be gone. It's so hard to improve your community when this keep happening to your servers.

Are the affected customers going to be reimbursed?

Okay one of my Second Assault servers just lasted 30 minutes, this is getting worse guys. Time to look for another GSP i guess. Can't be loyal while this is happening.

Update my second server crashed just minutes after the other one. *Sigh*

We just had our 4th crash since Second assualt. Can't get through 1 map.

massive issues tonight, my server had a ping of 124 when it dumped everyone. Clearly this is DDOS. Ausgamers - how can you mitigate this?

I have servers with Hypernia, and NFO (US based). Hypernia was also dumping players like a MOFO, clearly DDOS. NFO hasnt crashed, but laggy-ness can be an issue.

Brendan - we need help....

I'd love to get my hands on the people responsible for the DDoS. If only it were easy.

Yeah it would be awesome to be able to ban them from ever using the internet again ... oh and to make them pay for the downtime of everyone's servers. ;)

I wouldn't jump ship and do what we are doing, for us it was our last resort to keep our clan going as this has been happening to us for about 3x years now, and it's only ever let up for a few months at a time.... I was at the point of giving up.

Our USA based instances have gone down 4x times today from blaze disconnects, so it's by no means a perfect solution. It's probably a minimum of twice a day that this happens, sometimes it's within 30-40minutes of seeding the server again and it's totally random.

Can't say I've heard the rumours that the blaze back end is being DDoSed from any other GSP, I do have a message that states about 30% of server connections were dropped at a specific time today but that's about all, and it's not uncommon for this to occur.

Either way, BF server IPs are whitelisted, so it's not exactly hard to put filters in place to restrict connectivity only to those hosts, there's a few ways of doing this and all generally have minimal performance impact (boarder router)..

ICMP is often rate limited or set to a lower priority so a high ping doesn't necessarily relate to an attack. We had to disable our latency manager over all of our instances as users were constantly being kicked for high pings, users that generally had a ping of 5-10ms were spiking to 500ms+ at times and this wasn't unique to a specific ISP, this was when our servers were working.

Most of the attacks against our kit in the USA (forums/teamspeak) and our battlefield servers where I have access to view what's going on, or read the reports. Almost all of those attacks are NTP reflection attacks, with the occasional vulnerability attack (level 7 attacks). I would say 98-99% of these attacks are only 1-3Gbit at best, with the odd huge one coming through that can cause issues, so 99% of the time the attacks are mitigated and have no impact on our players.

(Disclaimer I don't work for ausgamers so this is speculation)

Granted I can't see someone hitting Ausgamers with 10-70Gbit DDoS attacks all day long, attacks of this size aren't as common as people might think, and if they were that large they'd be causing much bigger problems and you'd likely see IPs being null-routed , I've come to the conclusion that there's absolutely no mitigation equipment in place at all for the game servers.

Or the people configuring it have nfi what they are doing, either way we are never likely to find out.

Post up the graphs, it would go totally against what I've seen over the past fifteen years of my career of which two years was spent at Telstra, it also goes against what most security experts publish in terms of the average attack strength.

What equipment do you guys use to mitigate attacks? telling me if it's Arbor, NSFOCUS or X provider shouldn't be a problem.... I just get told it's all a big secret hence the speculation and doubt that anything is in place.

I've shown you mine before ;) I've also gone into detail about the equipment in place, in fact most of the time we post the information on the forum so our player base has an idea as to what's going on.

I've tried every GSP in AU and fed that information back to you with regards to performance and how we were attacked constantly, yet there seems to have been a fair bit of denial that it was happening here.

Either way, I'm sure I won't get any of the above questions answered, I just hope the issue gets resolved so I can come back and lodge more tickets just to annoy you!.

We have some changes in progress to respond to the current batch of DDOS attacks that we expect will restore a great deal of stability on our end, but we are aware of continued attacks against Blaze as well, which are beyond our control. I'm expecting an update on this this afternoon or tomorrow morning.

Hopefully on what im about to say won't jynx me or the rest of us. Servers only crashed once yesterday, and so far today from 11am, my servers haven't crashed at all. Keep up the good work Malthius and AusGamers!

Hopefully on what im about to say won't jynx me or the rest of us. Servers only crashed once yesterday, and so far today from 11am, my servers haven't crashed at all. Keep up the good work Malthius and AusGamers!

It's all your fault :)

We had a good run for a couple days,only a few disconnects.

Back to normal now.

One of my servers experienced 3 crashes today, looks like the problem is coming back

Brendan, I wasn't calling you a liar, at least that wasn't my intention... I understand what the situation is with regards to Ausgamers and Telstra and you've explained the level of access you guys have to the equipment in the past, so unless things have changed that's rather limited which probably makes it harder on you.

If Telstra is using BGP to redirect traffic to scrubbing appliances in the event of an attack it's not an appropriate solution for "gaming servers" especially of the bf4 nature granted you only need to look at them sideways and all 64 players drop off.

Well, when I say "DDOS stuff is being tweaked" and you say "there isn't any DDOS stuff", there is a pretty clear inference...

I can't confirm or deny what the hardware does, but what I will say is that it worked in the past - we were able to entirely prevent DNS reflection attacks from having any effect on server stability, so whatever we do have in place was entirely appropriate. It took a little time to get it tweaked to the point where it was protecting the servers - if we dial the knob too far to the left it won't protect the servers from all the attacks, and too far to the right and it sees legitimate traffic as an attack and disconnect the servers itself. (Funny example - I was staging the files for a patch deployment and triggered an alert due to traffic size, stopping me from staging the package).

We are now going through the same process for the NTP reflection attacks. Same as everyone else in the known universe.

I don't get to play with the big metal toys, no, but it is inconceivable that they are telling me that specific changes are being made while not actually doing anything. I give them too much hard data on player counts (which are polled every 10 seconds) and disconnections to try to do some sort of placebo thing.

Almost without fail we will have one each night at 23:50/Midnight.

Anyone else having the same issue?

ok we are up to 6 crashes on 4/8 of our servers already today and it is only 9am

i am just about over this completely! we have 8 servers with you guys and 3 with other providers (which do not crash nearly as regularly )
remind me, why are putting up with this and have been for 6 months ?

Last time I posted about this all my instances started going offline non stop... but here goes, my Ausgamers instance has only been offline once I believe in the last few days, mind you it's not really getting a huge workout either like it used to.

The new maps if you have them in your rotation are buggy as all hell and are causing server crashes, we were seeing 12+ a day on Caspian, Oman crashes a few times a day as well.

R28 update now so hopefully that makes things a little more stable, sorry couldn't help but laugh ;(

If you're going to waste your time calling someone at least make it worth your while and call Michelle...

Keep getting this (and everyone else), even after a restart....

Error Game disconnected: Client closed down connection. Check your network connection and make sure that port 3659 (UDP) is open.

Meh I gave up seeding mine, got it going a dozen or so times and it was taken offline each time, then couldn't even join the instance.

Same s*** that's been happening for a long arse time to us.

I pretty much have the "packetloss icon" on my screen non stop, same with other players, checking it out last night there was ~2% packetloss occurring on the server for an hour or so that I tested.

Unfortunately I can't check the path back....

Malthius. Are you hounding DICE for a fix? Im disconnecting like this going out of fashion.

I have tried to be patient for months, waiting for a fix. I acknowledge that Ausgamers are advising the problem rests with Dice however I can't keep a server populated for more than a couple hours lately and it is then black listed for days. Good luck to Ausgamers - I might be back if the stability is sorted in the future. For now, I'm done. Cancelled.

F5 DDoS Mitigation Technology

It beyond a joke we shouldnt pay full price for something we do not have.Its at the point where a ombudsman should be contacted if we get charged full price.

there f***ed i have 7 and they have stuffed up like 8 times today all of them.

I feel your pain.

.https://imagizer.imageshack.us/v2/1135x354q90/20/lr58.png

My gameservers instance is actually worse, I manage 1-2 rounds before it goes arse up and then wait ~40+ hours hours for some generic response and it costs $30 a month more than here.

The strange thing is no "lost connections" in the console or nothing, just 64 players disappearing into oblivion. At least on our Ausgamers instance we see all the players losing their connection and it's pretty much expected.

Off topic a bit but does everyone else have crap loads of rubber banding in-game?

I couldn't run Dragon here, US dedi no problem at all but not here 100% CPU on the graph and rubber banding galore.

SA maps here and our instance crashed all the time so we stopped it, tried it on GS twice and they dropped all players, US dedi didn't like Caspian but the rest worked fine so it's odd that there's so many different experiences when it's all of the same image.

Metro really isn't that popular any more, nothing like it used to be that's for sure.

Ok,

We now have our server disconnecting at 11 minutes past the hour every hour.

Started Wednesday 26th and was every 3 hours without fail.

Last night (friday 28) we had our normal 19:11 disconnect then 21:11,22:11,23:11.. etc..and been every hour since.

Anyone else having same issue?

Main server dropped all players x 3 today, 3 servers un contactable via web "Server 500 error" (hosted on different hosts) at 19:17 this evening.
203.46.105.22
203.46.105.24
203.46.105.72

Fix your network please, find where it broken and fix it. It aint rocket science.

There are many free network monitoring tools available that can pin point where the issue is. If it is DDOS please let us know don't hid it.

Main server dropped all players x 3 today, 3 servers un contactable via web "Server 500 error" (hosted on different hosts) at 19:17 this evening.
203.46.105.22
203.46.105.24
203.46.105.72

Fix your network please, find where it broken and fix it. It aint rocket science.

There are many free network monitoring tools available that can pin point where the issue is. If it is DDOS please let us know don't hid it.

Seems all of Ausgamers went down just now.

Back up for a few minutes then down again.

We have had 20 today so far...every hour to the minute...

Edit: 24 disconnects.

YOU HAVE NO FAVORITE SERVERS
Add servers to your favorites by clicking the star in the list.

Strange i had them a few minutes ago....


Oh wait they are all AusGamer servers.

Cant even get to the GameCreate Interface.

Yep same, I can't get to GameCreate either. :(

Big Lag spike affecting both 203.46.105.22 and 203.46.105.24 at 18:56

Anything to report AusGamers staff?

Lol i cant keep a server running for more than 30 mintues with the new DLC. Both servers keep crashing out. Weird since the past few days the server has been pretty good.

Hey I did tell everyone in one or two of these threads not to jump ship as the other GSPs are the same or worse ;)

It's interesting that you had a relatively positive experience with Hypernia as for us they were most definitely the worst GSP we've ever used in AU and we've used all of them multiple times over.

The reality is that the only GSP in AU that has a large network (owned/operated) behind them is Ausgamers, gameservers uses internodes/iinets and the rest are simply purchasing cheap arse xeon 1230s from dedicatedgaming or servers Australia.

While i3d does have a limited network presence in AU, it has next to no capacity which is why they run their servers off of dedicatedgamings network.

NFO has been an amazing experience, seriously I can't tell you how fantastic these guys are, if I ever did need anything changed they would respond to the ticket within two minutes (24/7) and 99% of the time fixed it.

On the DDoS side of things, when we moved to them our instances were all being attacked daily, I was able to see and capture this information, view it on the network graphs and they generally detected it and alerted me as well. I haven't actually had an attack on our US instances in a while now as the people doing it have given up on those instances and just focus on our AU based ones.

We will however be shutting our NFO dedicated down, $600+ a month on US and AU based instances is simply not sustainable and some of our clan members don't want to play on US based instances so we will be back to being AU based 100%. It's a shame as I was enjoying not having to restart instances all the time or coming online to the 30+ BL messages asking why the instance went down and their scores were lost.

Ok so the last 4 days has been horrific, we cant sustain a server more than an hour without a Blaze Disconnect.

Any thoughts if Ausgamers is actually still experiencing ddos, or "working with DICE" on a fix?

At the moment it seems the only option is to cancel the lot, as paying for something we can't use for more than an hour at a time is ludicrous.

Has anyone found a host here in OZ that is not having these problems. We are currently using Gameservers and our server is almost unusable and we can't get a bloody reply from support. We are thinking of moving but I'm buggered if I can work out who to move to.