pfSense slow site-to-site VPN
-
I'm fighting with this for quite some months now.
Site A:
- pfSense-latest running on an Intel Atom D525 (SuperMicro board) using Intel NICs.
- Wirespeed for plain transfers from the internet (e.g., downloading some Linux ISO)
Site B:
- Single Intel Xeon 2603v3 (just a testing machine)
- pfSense-latest running on top of Hyper-V 2012 R2
- 2 virtual NICs:
a) WAN with static VLAN assignment on the vNIC
b) Local nets with VLAN tagging from within the VM (due to the 8 pNIC limit in Windows/Hyper-V) - Near wirespeed for plain transfers from the internet (e.g., downloading some Linux ISO)
- Decent performance on traffic routed over the box (wirespeed), way faster when doing inter-VLAN between VMs on the same host (~ 300MB/s for large sequential files, the SSDs on the host just can't write faster)
Uplink between both sites is 1GB/s dark fiber at the moment, will be 10GB/s sometime soon. I'm seeing near wirespeed performance for plain traffic between both sites (FTP transfer for example), but Site-to-Site VPN using OpenVPN is capped at 7 MB/s. IPsec is a little bit faster, maybe 8-9 MB/s. Switches on both sites are externally managed Cisco 2960-X's. CPU usage during VPN transfers goes up to 20-30% at 7MB/s on Site A. At Site B, it's hard to even see an increased load. I don't expect wirespeed here, the little Atom is getting old, but at least like 20-40 MB/s. RTT between A and B is < 1.8ms (peak).
Why VPN over dark fiber? The line is running through a junction point in a public building I do not have control of.
Already did the usual stuff like limiting MTU, configuring sendbuf's and so on.
Any ideas? -
before continue reading your issue, i want to tell you that pfsense will not play well in virtual environment, in their official website too many people complaining about slow connection when installing pfsense in virtual environment,
-
i think it is a good idea to install the pfsense in SITE B in a physical machine
-
@IT-ADMIN said in pfSense slow site-to-site VPN:
i think it is a good idea to install the pfsense in SITE B in a physical machine
Even if done temporarily, it would allow you to take the HV out of the equation.
-
I'd try running the VPN on something other than the router. Try spinning up OpenVPN on a separate virtual machine and test it with that once.
You'd probably be better off using VyOS to do routing. Pfsense really does seem like a kludge after getting to know VyOS.
-
Two questions about the CPU load... is that total or is that for a single core? What is the load and load factor on that unit? It is very possible that the one core that OpenVPN can use for the SSL encoding is hitting 100% and there is no way to push it faster.
-
@travisdh1 said in pfSense slow site-to-site VPN:
You'd probably be better off using VyOS to do routing. Pfsense really does seem like a kludge after getting to know VyOS.
NTG Lab has a massive quad core Xeon, 12GB RAM, VyOS router heading to the LA Colocation America facility right now
-
Thanks everyone for all your suggestions, really appreciate that. Getting back a little late here, totally forgot that my wife planned a small trip for today
@IT-ADMIN: Thanks, already on my todo-list. Just had no machine at hand. I know about issues with pfSense running virtual, but as I said, performance is great when plain traffic gets routed over virtual and / or physical NICs. But you are right, should take virtualization out of the equation.
@Danp: Agree.
@travisdh1: Interessting, didn't thought about running OpenVPN standalone outside of pfSense. pfSense is known to be a bit... picky when it comes to OpenVPN. BTW: Im not a big fan of Vyatta. pfSense is running on my sites for nearly 10 years now. Would rather like to solve the problem
@scottalanmiller: Single core load measured over 5 mins outside of business hours.
Will test with a physical pfSense on some Intel NUC maybe. If that does not work, I could still do some tests with a standalone OpenVPN on some Linux host. Will report back later.
-
okay, single core that's not too bad. I'd still check the load numbers (uptime will give you this quickly) and see if there is some wait state catching you.
-
@scottalanmiller good point, thanks
-
@IT-ADMIN said in pfSense slow site-to-site VPN:
before continue reading your issue, i want to tell you that pfsense will not play well in virtual environment, in their official website too many people complaining about slow connection when installing pfsense in virtual environment,
That used to be the case, newest versions are fine.
-
Why Open VPN for site to Site over IPSEC? OpenVPN is normally much slower..
OpenVPN was more made for easy configuration.
-
Also have you considered TINC full mesh on Pfsense, I have used it in the past when I worked at smaller companies.
-
@Jason said in pfSense slow site-to-site VPN:
OpenVPN is normally much slower..
No preference for OpenVPN, tried both, IPsec being just 1MB/s faster. Oddly, latency is still great while throughput is low. Both lines are sync 1GB/s, just some fiber and roughly 4 km / 2.5 miles in between. Next to no reflections on the fibre.
Async lines are a known problem especially in OpenVPN, but that shouldn't be the case
-
@Jason Not yet. Problem is, there's some confidential data going over that wire. Sure, already encrypted on OSI-7, but I don't feel comfortable using a solution in this context I don't know. Would rather like to stick to IPsec (preferred) or OpenVPN.
-
@Jason said in pfSense slow site-to-site VPN:
@IT-ADMIN said in pfSense slow site-to-site VPN:
before continue reading your issue, i want to tell you that pfsense will not play well in virtual environment, in their official website too many people complaining about slow connection when installing pfsense in virtual environment,
That used to be the case, newest versions are fine.
Does it have built in PV drivers for most platforms? Which ones does it support well?
-
@Jason said in pfSense slow site-to-site VPN:
Why Open VPN for site to Site over IPSEC? OpenVPN is normally much slower..
Specifically in CPU usage.
-
Try this: https://forum.pfsense.org/index.php?topic=47567.0
What's your protocol set to? TCP or UDP?
-
@marcinozga Thanks, but already tried net.inet.ip.fastforwarding in all combinations with TCP and UDP.
