What's new

Packet Loss On RCRP

Status
Not open for further replies.

Max Bluman

Donator
Joined
Apr 1, 2016
Messages
282
fr0st said:
Max Bluman said:
fr0st said:
Max Bluman said:
Michael. said:
It's not your problem it's probably the /host/ problem.
First of all, why is it a host problem?
Answer: Well, all the players like legit /ALL/ players are lagging that's not their net problem.
Second of All, Ok fine we understand it's a host problem but why do they lag?
It's the location in-which the host is bought from [for eg: France, London, other countries.]
Third of All, I'm not just talking about a windows host this problem will also show itself in VPS Host's too.
RC-RP you got to change your Hosting Service, contact me for more info, I can guide in-getting good services. (I've ran many servers myself, LINUX - WINDOWS) (This problem always has been my VPS or Host problem but It was fixed after I once tried OVH Linux Cloud Hosting or Ultra-host for windows.and they ran perfectly fine. I may be happy if this is the /answer/ to your situation, have fun fella's. :cowboy:
Sounds to me that you are quite literally talking out of your ass. OVH uses cheap bandwidth and oversatured bandwidth so packet loss from time to time will make sense, also for example the server receives a DDoS attack and bypasses OVH VAC. Tommy needs to go with a provider that provides a premium blend of bandwidth, t1 providers. Perhaps somebody like Zare, Clouvider or another provider within Germany/Netherlands.
Some of what you are saying isn't far from the truth, but I think you lack a few key points to understand why OVH is superior and why this problem even exists in the first place.

I'll address your statements and then explain some more.

Max Bluman said:
OVH uses cheap bandwidth and oversatured bandwidth so packet loss from time to time will make sense
I don't know where you got the information about "cheap" bandwidth, but they infact pay for more bandwidth than many other providers because of their DDoS protection. For example, in Europe, they have a 140Gbps circuit with Cogent, A 230Gbps circuit with Level3, and a 310Gbps circuit with TATA.

Yes, there's times where a specific peer can get overloaded from the DDoS attacks and cause packet loss for anyone who takes that routing path, but bgp is more intelligent nowdays and has failover. So as soon as a specific peer starts flapping, they'll pull the advertised routes from them and another peer will have a higher weight.

Max Bluman said:
also for example the server receives a DDoS attack and bypasses OVH VAC.
OVH's "vacuum" system only kicks in once an attack has been detected, so what you're saying is reversed. There's also a firewall setting to enable permanent mitigation so that it goes through their scrubbing center at all times, but we do not use this as their scrubbing centers have added latency and packet loss.

Max Bluman said:
Tommy needs to go with a provider that provides a premium blend of bandwidth, t1 providers. Perhaps somebody like Zare, Clouvider or another provider within Germany/Netherlands.
As I mentioned above, OVH peers with the majority of the public internet exchanges and carriers in Europe and North America. Some of their Tier-1 providers are Cogent, Tata, Level3, Global Crossing, Seabone, NTT, Telia, Opentransit, T-System, Equinix (San Jose/Dallas/Ashburn/etc). So I'm sorry to tell you this, but peering isn't the problem.


Now time for a little history! RC-RP was hosted by Snelserver in the past which was in Rotterdam, NL. They had great servers and a great network since they had a colocation with i3d. (i3d hosts some stuff for Sony and other gaming services)

The problem was always DDoS issues. We had a full 1Gbps link with Snelserver, but it never mattered. Someone with a booter would always be able to saturate the connection to the point where nobody could query/use the server. So after many packet captures, DDoS analysis, pattern analysis, etc. I had Snelserver install firewall rules to their edge network to block stuff like fragmented packets, chargen, etc. It all worked well for about a few weeks until some new attack vector came out (like DNS/NTP reflection/amplification, etc) and we were back into the same boat again. We had also paid 30 euro per each rule on the edge network, so I think you can see where this is going.

This is what the dark ages looked like for RC-RP:


It wasn't affordable to continually pay 30 euro for every firewall rule, and it wasn't feasible to keep up with every attack around the clock. (we're not a team of securiy researchers!) So we looked for alternatives and OVH seemed to get good praise. After testing of my own and extensive configuration of the firewall, I found that their firewall was far superior, as was their pricing. (It was much cheaper than Snelserver, even had better server specs.)

So to conclude the reason for the switch to OVH in a laundy list;
1) Cheaper server, far better specs (CPU/RAM/HDD/BANDWIDTH)
2) Superior firewall, free to add/remove rules at any time, can disable/enable firewall rules at any time
3) Naturally DDoS resistant infastructure with scrubbing centers, anycast BGP, etc.
4) Better control panel features, abilites to auto-route IP addresses to different dedicated servers, ability to easily transfer IP's between servers.
5) Cheaper IP addresses, + only one time fee to add more IP's (1 euro)

And to finish off this post, I'd like to remind you that SA-MP still uses UDP for transferring packets to/from players. UDP is faster than TCP, but it is a natrually lossy protocol. It has no inherent ability to resend lost packets. So it is not resistant to packet loss like TCP is. Sure, both have their advantages and disadvantages, but loss is a clear disadvantage of UDP. So combine that with the fact that some people might be playing on WiFi, have family using Netflix, or actually use 1990's Sattelite internet (I'm looking at you Giga!), then that leads to the problem everyone describes simply as: "lag"
I am happy you took the time to write a reply to my statements the way you did, I didn't expect it.

I don't think you understand what I meant by "premium" bandwidth and I also don't think you know the difference between T1/T2 and T3 providers. OVH as a company only peers with Tata, Telia, Level 3, Telecom Italy, Cogent and HE (Hurricane Electric), I am unsure where you are getting all the other ones from but they simply aren't bandwidth providers directly with OVH. Out of all of these providers, the only two premium bandwidth providers are L3 and Tata. The rest are a mix of Tier 3 & 2 providers, for example HE and Cogent are both Tier 2. This is practically a guarantee that they sell their bandwidth cheaper, meaning an oversold network and not a very good setup at all. This can cause increased latency to some people, and sometimes packet loss if one of these peers/transit providers are being hit with a gargantuan DDoS Attack for example, however I don't think that was the issue in this instance.

I agree with you that OVH has an absolutely massive amount of bandwidth, this is expected with a company that hosts more than 310K+ physical servers, not including virtual machines/websites/email servers etc. However I am sure you are aware how the DDoS Protection works, it's several in-line devices performing DPI on each packet to determine what needs to be blocked/re-routed to scrubbing center and what needs to be allowed past. These systems are built to self learn and automatically create firewall rules, like what you used to pay 30 EUR per rule for. However, sometimes people with more technological knowledge and aren't just using some booter they found on Google end up modifying packet payloads that aren't normal and won't be picked up by these devices hence bypassing OVH VAC system. Certain methods like MSSQL, TCP ACK used to bypass it all the time but they have since blocked these methods.

I have VERY rarely experienced packet loss issues with OVH, usually their service is solid and has good uptime. The fact all of these people are having packet loss and different times, hints to me that somebody is attacking the server with one of these attack methods that isn't being picked up by VAC. Do you have a different opinion as to what is currently causing the issue?
I quoted their website directly as companies they listed as "Tier 1"; https://www.ovh.com/us/dedicated-server ... etails.xml

I know the difference between the tiers and I didn't think it was relevant to go into an indepth discussion of that on this topic. Just see the above link for what I referenced.

I cannot remmeber the last time someone actually took down the server entirely from a DDoS attack. So the firewall is working as intended and nobody is bypassing it with any special crafted attacks. Otherwise it would be everyone affected simultaneously instead of random people having it off and on. I don't really have any opinion on what it might be at the moment.

I can say for certain that the packet loss issues flared up more once we switched to OVH, but ask yourself; Would you rather have a server that works smoothly until it goes down entirely (which was frequently and for many hours at a time), or have a server that has almost 100% uptime, but ocassionally has packet loss or desync?

I'd obviously choose the second, even if it means degrading the performance but maintaining better uptime and availability.

We're still looking at what can be done to fix this, but if it ends up being a peering/routing problem, it's usually out of our hands.
I wonder if they just haven't updated that in about 9 years, you can find proper peers below. I highly doubt it's a peering/routing problem but I am just saying you could provide a better connection to all of your users by going with a more premium provider (some of which I listed before). I haven't even experienced this packet loss issue myself, just providing my 2 cents on possibilities :detective:

https://bgp.he.net/AS16276
 

fr0st

SysAdmin
System Administrator
Joined
Jan 28, 2011
Messages
442
Location
Texas
Max Bluman said:
fr0st said:
Max Bluman said:
fr0st said:
Max Bluman said:
Sounds to me that you are quite literally talking out of your ass. OVH uses cheap bandwidth and oversatured bandwidth so packet loss from time to time will make sense, also for example the server receives a DDoS attack and bypasses OVH VAC. Tommy needs to go with a provider that provides a premium blend of bandwidth, t1 providers. Perhaps somebody like Zare, Clouvider or another provider within Germany/Netherlands.
Some of what you are saying isn't far from the truth, but I think you lack a few key points to understand why OVH is superior and why this problem even exists in the first place.

I'll address your statements and then explain some more.

Max Bluman said:
OVH uses cheap bandwidth and oversatured bandwidth so packet loss from time to time will make sense
I don't know where you got the information about "cheap" bandwidth, but they infact pay for more bandwidth than many other providers because of their DDoS protection. For example, in Europe, they have a 140Gbps circuit with Cogent, A 230Gbps circuit with Level3, and a 310Gbps circuit with TATA.

Yes, there's times where a specific peer can get overloaded from the DDoS attacks and cause packet loss for anyone who takes that routing path, but bgp is more intelligent nowdays and has failover. So as soon as a specific peer starts flapping, they'll pull the advertised routes from them and another peer will have a higher weight.

Max Bluman said:
also for example the server receives a DDoS attack and bypasses OVH VAC.
OVH's "vacuum" system only kicks in once an attack has been detected, so what you're saying is reversed. There's also a firewall setting to enable permanent mitigation so that it goes through their scrubbing center at all times, but we do not use this as their scrubbing centers have added latency and packet loss.

Max Bluman said:
Tommy needs to go with a provider that provides a premium blend of bandwidth, t1 providers. Perhaps somebody like Zare, Clouvider or another provider within Germany/Netherlands.
As I mentioned above, OVH peers with the majority of the public internet exchanges and carriers in Europe and North America. Some of their Tier-1 providers are Cogent, Tata, Level3, Global Crossing, Seabone, NTT, Telia, Opentransit, T-System, Equinix (San Jose/Dallas/Ashburn/etc). So I'm sorry to tell you this, but peering isn't the problem.


Now time for a little history! RC-RP was hosted by Snelserver in the past which was in Rotterdam, NL. They had great servers and a great network since they had a colocation with i3d. (i3d hosts some stuff for Sony and other gaming services)

The problem was always DDoS issues. We had a full 1Gbps link with Snelserver, but it never mattered. Someone with a booter would always be able to saturate the connection to the point where nobody could query/use the server. So after many packet captures, DDoS analysis, pattern analysis, etc. I had Snelserver install firewall rules to their edge network to block stuff like fragmented packets, chargen, etc. It all worked well for about a few weeks until some new attack vector came out (like DNS/NTP reflection/amplification, etc) and we were back into the same boat again. We had also paid 30 euro per each rule on the edge network, so I think you can see where this is going.

This is what the dark ages looked like for RC-RP:


It wasn't affordable to continually pay 30 euro for every firewall rule, and it wasn't feasible to keep up with every attack around the clock. (we're not a team of securiy researchers!) So we looked for alternatives and OVH seemed to get good praise. After testing of my own and extensive configuration of the firewall, I found that their firewall was far superior, as was their pricing. (It was much cheaper than Snelserver, even had better server specs.)

So to conclude the reason for the switch to OVH in a laundy list;
1) Cheaper server, far better specs (CPU/RAM/HDD/BANDWIDTH)
2) Superior firewall, free to add/remove rules at any time, can disable/enable firewall rules at any time
3) Naturally DDoS resistant infastructure with scrubbing centers, anycast BGP, etc.
4) Better control panel features, abilites to auto-route IP addresses to different dedicated servers, ability to easily transfer IP's between servers.
5) Cheaper IP addresses, + only one time fee to add more IP's (1 euro)

And to finish off this post, I'd like to remind you that SA-MP still uses UDP for transferring packets to/from players. UDP is faster than TCP, but it is a natrually lossy protocol. It has no inherent ability to resend lost packets. So it is not resistant to packet loss like TCP is. Sure, both have their advantages and disadvantages, but loss is a clear disadvantage of UDP. So combine that with the fact that some people might be playing on WiFi, have family using Netflix, or actually use 1990's Sattelite internet (I'm looking at you Giga!), then that leads to the problem everyone describes simply as: "lag"
I am happy you took the time to write a reply to my statements the way you did, I didn't expect it.

I don't think you understand what I meant by "premium" bandwidth and I also don't think you know the difference between T1/T2 and T3 providers. OVH as a company only peers with Tata, Telia, Level 3, Telecom Italy, Cogent and HE (Hurricane Electric), I am unsure where you are getting all the other ones from but they simply aren't bandwidth providers directly with OVH. Out of all of these providers, the only two premium bandwidth providers are L3 and Tata. The rest are a mix of Tier 3 & 2 providers, for example HE and Cogent are both Tier 2. This is practically a guarantee that they sell their bandwidth cheaper, meaning an oversold network and not a very good setup at all. This can cause increased latency to some people, and sometimes packet loss if one of these peers/transit providers are being hit with a gargantuan DDoS Attack for example, however I don't think that was the issue in this instance.

I agree with you that OVH has an absolutely massive amount of bandwidth, this is expected with a company that hosts more than 310K+ physical servers, not including virtual machines/websites/email servers etc. However I am sure you are aware how the DDoS Protection works, it's several in-line devices performing DPI on each packet to determine what needs to be blocked/re-routed to scrubbing center and what needs to be allowed past. These systems are built to self learn and automatically create firewall rules, like what you used to pay 30 EUR per rule for. However, sometimes people with more technological knowledge and aren't just using some booter they found on Google end up modifying packet payloads that aren't normal and won't be picked up by these devices hence bypassing OVH VAC system. Certain methods like MSSQL, TCP ACK used to bypass it all the time but they have since blocked these methods.

I have VERY rarely experienced packet loss issues with OVH, usually their service is solid and has good uptime. The fact all of these people are having packet loss and different times, hints to me that somebody is attacking the server with one of these attack methods that isn't being picked up by VAC. Do you have a different opinion as to what is currently causing the issue?
I quoted their website directly as companies they listed as "Tier 1"; https://www.ovh.com/us/dedicated-server ... etails.xml

I know the difference between the tiers and I didn't think it was relevant to go into an indepth discussion of that on this topic. Just see the above link for what I referenced.

I cannot remmeber the last time someone actually took down the server entirely from a DDoS attack. So the firewall is working as intended and nobody is bypassing it with any special crafted attacks. Otherwise it would be everyone affected simultaneously instead of random people having it off and on. I don't really have any opinion on what it might be at the moment.

I can say for certain that the packet loss issues flared up more once we switched to OVH, but ask yourself; Would you rather have a server that works smoothly until it goes down entirely (which was frequently and for many hours at a time), or have a server that has almost 100% uptime, but ocassionally has packet loss or desync?

I'd obviously choose the second, even if it means degrading the performance but maintaining better uptime and availability.

We're still looking at what can be done to fix this, but if it ends up being a peering/routing problem, it's usually out of our hands.
I wonder if they just haven't updated that in about 9 years, you can find proper peers below. I highly doubt it's a peering/routing problem but I am just saying you could provide a better connection to all of your users by going with a more premium provider (some of which I listed before). I haven't even experienced this packet loss issue myself, just providing my 2 cents on possibilities :detective:

https://bgp.he.net/AS16276
OVH's websites/different branches are a mess, so I'm not surprised. The problem with going with a more "premium" bandwidth provider is that they're less resistant to DDoS attacks like OVH is. Clouvider for instance would likely nullroute the IP for 24 hours after an attack rolled in. I don't know their specific policies, but I can guess they would not let you configure the firewall on your own. So we'd be back at the Snelserver crossroads again where every attack requires analysis, 30 euro per rule, and additional downtime.

Ofcourse we could just get a 10Gbps uplink at a better provider and put a hardware firewall in front of it, but can you guess how much that would cost at any provider? The answer is: More than we're willing to pay considering our current pricepoint at OVH.
 

Max Bluman

Donator
Joined
Apr 1, 2016
Messages
282
OVH's websites/different branches are a mess, so I'm not surprised. The problem with going with a more "premium" bandwidth provider is that they're less resistant to DDoS attacks like OVH is. Clouvider for instance would likely nullroute the IP for 24 hours after an attack rolled in. I don't know their specific policies, but I can guess they would not let you configure the firewall on your own. So we'd be back at the Snelserver crossroads again where every attack requires analysis, 30 euro per rule, and additional downtime.

Ofcourse we could just get a 10Gbps uplink at a better provider and put a hardware firewall in front of it, but can you guess how much that would cost at any provider? The answer is: More than we're willing to pay considering our current pricepoint at OVH.
Clouvider operates two methods of DDoS protection. 100% in-line DDoS Protection appliance (no scrubbing centers, 0.1ms increase) in which they do not confirm the amount it can handle but have confirmed privately with me it's 40Gbps+. If something then outmatches that, they raise it with NTT (one of their ip transit providers) and it is mitigated by their huge capacity. A 10Gbps port would be useless in this day and age, the average DDoS Attack is like 15-25Gbps. I would honestly look into a few hosts and see if you can get yourself a better deal, OVH isn't particularly the best anymore, a lot of hosts have caught up :read:
 

fr0st

SysAdmin
System Administrator
Joined
Jan 28, 2011
Messages
442
Location
Texas
Max Bluman said:
OVH's websites/different branches are a mess, so I'm not surprised. The problem with going with a more "premium" bandwidth provider is that they're less resistant to DDoS attacks like OVH is. Clouvider for instance would likely nullroute the IP for 24 hours after an attack rolled in. I don't know their specific policies, but I can guess they would not let you configure the firewall on your own. So we'd be back at the Snelserver crossroads again where every attack requires analysis, 30 euro per rule, and additional downtime.

Ofcourse we could just get a 10Gbps uplink at a better provider and put a hardware firewall in front of it, but can you guess how much that would cost at any provider? The answer is: More than we're willing to pay considering our current pricepoint at OVH.
Clouvider operates two methods of DDoS protection. 100% in-line DDoS Protection appliance (no scrubbing centers, 0.1ms increase) in which they do not confirm the amount it can handle but have confirmed privately with me it's 40Gbps+. If something then outmatches that, they raise it with NTT (one of their ip transit providers) and it is mitigated by their huge capacity. A 10Gbps port would be useless in this day and age, the average DDoS Attack is like 15-25Gbps. I would honestly look into a few hosts and see if you can get yourself a better deal, OVH isn't particularly the best anymore, a lot of hosts have caught up :read:
Their solution doesn't accommodate my need for custom firewall rules, though. I have a lot of custom rules on OVH (and different ones for different IP's) So unless their network has fully automatic mitigation like OVH does, then I don't think that would work for us.

If you happen to have a server with Clouvider that can be used for testing, please PM me.
 

Max Bluman

Donator
Joined
Apr 1, 2016
Messages
282
fr0st said:
Max Bluman said:
OVH's websites/different branches are a mess, so I'm not surprised. The problem with going with a more "premium" bandwidth provider is that they're less resistant to DDoS attacks like OVH is. Clouvider for instance would likely nullroute the IP for 24 hours after an attack rolled in. I don't know their specific policies, but I can guess they would not let you configure the firewall on your own. So we'd be back at the Snelserver crossroads again where every attack requires analysis, 30 euro per rule, and additional downtime.

Ofcourse we could just get a 10Gbps uplink at a better provider and put a hardware firewall in front of it, but can you guess how much that would cost at any provider? The answer is: More than we're willing to pay considering our current pricepoint at OVH.
Clouvider operates two methods of DDoS protection. 100% in-line DDoS Protection appliance (no scrubbing centers, 0.1ms increase) in which they do not confirm the amount it can handle but have confirmed privately with me it's 40Gbps+. If something then outmatches that, they raise it with NTT (one of their ip transit providers) and it is mitigated by their huge capacity. A 10Gbps port would be useless in this day and age, the average DDoS Attack is like 15-25Gbps. I would honestly look into a few hosts and see if you can get yourself a better deal, OVH isn't particularly the best anymore, a lot of hosts have caught up :read:
Their solution doesn't accommodate my need for custom firewall rules, though. I have a lot of custom rules on OVH (and different ones for different IP's) So unless their network has fully automatic mitigation like OVH does, then I don't think that would work for us.

If you happen to have a server with Clouvider that can be used for testing, please PM me.
I don't currently have a node with Clouvider as my hardware is colocated with Serverius (I plan on moving to them by end of the year). I have asked Dominik if he can provide a test virtual machine for me as a network test that I'll just give to you if he does.

Will send ya a PM if I do get it, and yes, it is a fully automatic mitigation system like OVH.
 

Supernova

Silver Member
Joined
Mar 8, 2017
Messages
1,256
Location
Algeria
I just logged in and saw my packet loss was 5 percent even though I have a low ping, seems like its back again.
 

Crazy Andre

Retired Admin
Joined
May 10, 2010
Messages
8,632
Supernova said:
I just logged in and saw my packet loss was 5 percent even though I have a low ping, seems like its back again.
It will keep coming and going. I started having PL issues around mid summer and its been on and off ever since. I thought originally it was due to player base but PL on this server is so inconsistent. This host sucks ass for me and its super frustrating no offense but I'm fed up with it. I never had PL issues this bad ever in my time on RCRP since 2010.
 

Crazy Andre

Retired Admin
Joined
May 10, 2010
Messages
8,632
My PL issue been acting up again to the point where I can't even click onto my player when logging in. :roll:
 

B3TT

Silver Member
Joined
Jan 23, 2017
Messages
658
Location
Pakistan.
I am in-game right now and tbh. I cant play today. Packetloss keeps on rising, game keeps lagging/desyncing and stuff.. Everything else is superfine except rcrp.

Sent from my WAS-LX1A using Tapatalk
 
Status
Not open for further replies.
Top