Problems with OpenVPN when server is FreeBSD 14.2 and client is 14.1

After replacing an existing gateway with new hardware, the OpenVPN connections had issues. The clients would connect, but non-trivial amounts of traffic would cause errors.

This post contains a workaround, by upgrading the OS. I would like to know the cause.

The symptoms

Running scp would give (my post on freebsd-net mailing list):

[12:33 mydev dvl ~] % scp dan@zuul.vpn:FreeBSD-14.2-RELEASE-amd64-memstick.img .
FreeBSD-14.2-RELEASE-amd64-memstick.img                         0%    0     0.0KB/s   --:--
ETAFssh_ssh_dispatch_run_fatal: Connection to 10.80.0.66 port 22: message authentication code incorrect
scp: Connection closed

Running a Bacula backup would result in:

SD says - Error: openssl.c:108 TLS read/write failure.: ERR=error:0A000119:SSL \
routines::decryption failed or bad record mac FD says - Error: bsock.c:397 Wrote \
43011 bytes to Storage daemon:bacula-sd-04.int.unixathome.org:9103, but only 0 \
accepted. SD says - Fatal error: append.c:327 Network error reading from FD. \
ERR=Unknown error: 9919

I posted the above to the FreeBSD users mailing list.

I also consulted the OpenVPN IRC channel on Libera.Chat.

MTU seemed to be the central theme amongst the suggestions. Changes did not seem to help.

After about a week of frustrations, I began to think it was some magic in the OpenVPN configuration which I was missing. The previous implementation worked reliably. It must be something I was doing.

I confirmed that scp host-to-host without the VPN worked as expect. I could copy over a 150GB file without error. This told me it was not the network and it wasn’t the hardware. Unless OpenVPN was using something non-VPN traffic was using.

After mentioning the scp problem on the Libera.Chat FreeBSD channel, mzar emailed me regarding my freebsd-net post. FreeBSD PR 276838 was mentioned. Based on his reply, I tried tun-mtu 1400″ on the service side. It did not solve the issue.

Looking over the clients

Until this point, I was concentrating my efforts on one client (r720-02). ON 2025-03-08, I tried the scp test on all 6 clients. One client (nagios03) worked, the the other 5 failed.

What was different with nagios03? It was on FreeBSD 14.2; the others were on FreeBSD 14.1 – would an upgrade fix this issue?

I upgraded tallboy. Problem went away. scp succeeded.

Then I upgraded two more hosts: aws-1 and zuul. The scp test now worked.

I decided to not upgrade two OpenVPN clients. They are still on FreeBSD 14.1 – I’ll keep them that way so I can do more testing should theories arrive regarding the cause of the problem.

What is the cause?

Why does upgrading the OS fix this problem?

I have two questions about the old gateway (answers added on 2025-03-10):

what version of FreeBSD was installed – FreeBSD 14.0-CURRENT amd64 1400094
what version of OpenVPN was used? – openvpn-2.6.8_1

The new gateway has tried to copy the configuration as much as I can. Some features were not implemented. Specifically:

up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'bast.example.org' 1"

I did implement a very simple solution for:

client-connect    /usr/local/sbin/serverlocal-events.sh
client-disconnect /usr/local/sbin/serverlocal-events.sh

How simple? This simple:

[13:18 gw01 dvl ~] % cat /usr/local/sbin/serverlocal-events.sh
#!/bin/sh

# Taken from https://forums.openvpn.net/viewtopic.php?t=43899

# Executed on the server side for client connect and disconnect events.

# Log client connect or disconnect event with IP address

case "$script_type" in
  client-connect)
    logger "$common_name connected with IP $trusted_ip"
    ;;

  client-disconnect)
    logger "$common_name disconnected with IP $trusted_ip"
    ;;

esac

exit 0

All it does is log. I’m sure it does nothing else useful. I’ll keep it though, just to keep track of connects & disconnects.

For a while, I was convinced there was something missing in my openvpn configuration which I needed. I was sure that was the cause of the problem and I could find it.

Future work for me

Try DCO: https://www.freebsd.org/status/report-2022-04-2022-06/if_ovpn/

It seems like most of my VPN nodes hav hardware encryption capabilities.

The OpenVPN server

[12:39 gw01 dvl ~] % grep -i AES /var/run/dmesg.boot
Features2=0x4ff8ebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,
x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
GEOM_ELI: Encryption: AES-XTS 128

Four of the OpenVPN clients:

aws-1

[12:37 aws-1 dan ~] % grep -i AES /var/run/dmesg.boot
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,
XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>

nagios03

[12:37 nagios03 dvl ~] % grep -i AES /var/run/dmesg.boot Features2=0xfeda3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,
AVX,F16C,RDRAND,HV>
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>

r720-02

[12:37 r720-02 dvl ~] % grep -i AES /var/run/dmesg.boot
Features2=0x1fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,
SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>

x8dtu

[12:37 x8dtu dvl ~] % grep -i AES /var/run/dmesg.boot
Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,
SSE4.1,SSE4.2,POPCNT,AESNI>
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>

Two clients do not:

tallboy

[12:37 tallboy dvl ~] % grep -i AES /var/run/dmesg.boot
aesni0: No AES or SHA support.
aesni0: No AES or SHA support.

zuul

[12:37 zuul dan ~] % grep -i AES /var/run/dmesg.boot
aesni0: No AES or SHA support.
aesni0: No AES or SHA support.