How to configure VMs running Netagent for high throughput production traffic
- Updated on Feb 19, 2022
- Production Tuning
Modern Linux systems can handle thousands (or millions) of connections, but the system default settings are often inadequate for such high throughput.
This guide provides our recommended settings for moderate to heavy usage. The settings can be increased beyond our recommendations for extremely heavy workloads. Please consult our support team for advice.
The file descriptor and conntrack settings are required for production use. The additional TCP stack settings are optional for most systems.
These recommendations apply to all Netagents, whether running in Host Agent mode or Access Tier mode.
A busy server uses a high number of file descriptors for incoming and outgoing connections. The default for most user accounts is only 1024 descriptors. You can check the current limit for your account with this command:
$ ulimit -n 8192
To increase the limit beyond the default, create a file called
/etc/security/limits.d/banyan.conf and add these lines to it.
* soft nofile 100000 * hard nofile 100000
If your system does not have a
limits.d directory, append the above lines to the bottom of
ulimit command to confirm that the new limits have taken effect.
$ ulimit -n 100000
There is also a system-wide limit that may need to be increased.
$ cat /proc/sys/fs/file-max 94158
If the output is less than 100000, adjust it by creating a file in
90-banyan.conf and add the following to it:
fs.file-max = 100000
Reboot for the change to take effect, or change the limit temporarily with this command:
sysctl -w fs.file-max=100000
Confirm the change by repeating the
$ cat /proc/sys/fs/file-max 100000
If you are using RedHat, CentOS, Fedora, or Scientific Linux, you may need to add
session required pam_limits.so to
/etc/pam.d/login. Consult your OS documentation.
Kernel conntrack parameters
Netagent uses a kernel module called
nf_conntrack to help track connections to backend services. Connections are tracked in a kernel hash table, and if the hash table fills up, the module will drop incoming packets.
If you see messages in the kernel log like these, you need to increase the hash table size:
[34625.043999] net_ratelimit: 29 callbacks suppressed [34625.044003] nf_conntrack: table full, dropping packet
To increase the size of the hash table, create a file in
banyan.conf and add the line:
options nf_conntrack hashsize=65536
If you do not have a
modprobe.d directory on your system, you can add the above line to the end of
This will adjust the size of the hash table to 64K entries. It takes effect on the next reboot. If you cannot reboot the system right away, you can also adjust the settings temporarily with these commands:
$ echo 65536 > /proc/sys/net/netfilter/nf_conntrack_buckets $ echo 262144 > /proc/sys/net/netfilter/nf_conntrack_max
Confirm the settings have taken effect:
$ cat /proc/sys/net/netfilter/nf_conntrack_buckets 65536 $ cat /proc/sys/net/netfilter/nf_conntrack_max 262144
Kernel TCP parameters
Here are some optional kernel tunables that should suffice for even very busy systems. Add these to the
/etc/sysctl.d/90-banyan.conf file you created above:
net.ipv4.tcp_rmem = 4096 25165824 25165824 net.ipv4.tcp_wmem = 4096 65536 25165824 net.core.somaxconn = 100000 net.ipv4.tcp_max_syn_backlog = 100000 net.core.netdev_max_backlog = 100000 net.core.rmem_max = 25165824 net.core.rmem_default = 25165824 net.core.wmem_max = 25165824 net.core.wmem_default = 65536 net.core.optmem_max = 25165824