• Production Tuning
  • Background
  • Recommended settings
    • File descriptors
  • Kernel conntrack parameters
  • Kernel TCP parameters

Production Tuning

Background

Modern Linux systems can handle thousands (or millions) of connections, but the system default settings are often inadequate for such high throughput.

This guide provides our recommended settings for moderate to heavy usage. The settings can be increased beyond our recommendations for extremely heavy workloads. Please consult our support team for advice.

The file descriptor and conntrack settings are required for production use. The additional TCP stack settings are optional for most systems.

These recommendations apply to all Netagents, whether running in Host Agent mode or Access Tier mode.

Recommended settings

File descriptors

A busy server uses a high number of file descriptors for incoming and outgoing connections. The default for most user accounts is only 1024 descriptors. You can check the current limit for your account with this command:

$ ulimit -n
8192

To increase the limit beyond the default, create a file called /etc/security/limits.d/banyan.conf and add these lines to it.

* soft nofile 100000
* hard nofile 100000

If your system does not have a limits.d directory, append the above lines to the bottom of /etc/security/limits.conf instead.

Repeat the ulimit command to confirm that the new limits have taken effect.

$ ulimit -n
100000

There is also a system-wide limit that may need to be increased.

$ cat /proc/sys/fs/file-max
94158

If the output is less than 100000, adjust it by creating a file in /etc/sysctl.d called 90-banyan.conf and add the following to it:

fs.file-max = 100000

Reboot for the change to take effect, or change the limit temporarily with this command:

sysctl -w fs.file-max=100000

Confirm the change by repeating the cat command.

$ cat /proc/sys/fs/file-max
100000

If you are using RedHat, CentOS, Fedora, or Scientific Linux, you may need to add session required pam_limits.so to /etc/pam.d/login. Consult your OS documentation.

Kernel conntrack parameters

Netagent uses a kernel module called nf_conntrack to help track connections to backend services. Connections are tracked in a kernel hash table, and if the hash table fills up, the module will drop incoming packets.

If you see messages in the kernel log like these, you need to increase the hash table size:

[34625.043999] net_ratelimit: 29 callbacks suppressed  
[34625.044003] nf_conntrack: table full, dropping packet

To increase the size of the hash table, create a file in /etc/modprobe.d called banyan.conf and add the line:

options nf_conntrack hashsize=65536

If you do not have a modprobe.d directory on your system, you can add the above line to the end of /etc/modprobe.conf instead.

This will adjust the size of the hash table to 64K entries. It takes effect on the next reboot. If you cannot reboot the system right away, you can also adjust the settings temporarily with these commands:

$ echo 65536 > /proc/sys/net/netfilter/nf_conntrack_buckets
$ echo 262144 > /proc/sys/net/netfilter/nf_conntrack_max

Confirm the settings have taken effect:

$ cat /proc/sys/net/netfilter/nf_conntrack_buckets
65536
$ cat /proc/sys/net/netfilter/nf_conntrack_max
262144

Kernel TCP parameters

Here are some optional kernel tunables that should suffice for even very busy systems. Add these to the /etc/sysctl.d/90-banyan.conf file you created above:

net.ipv4.tcp_rmem = 4096 25165824 25165824
net.ipv4.tcp_wmem = 4096 65536 25165824
net.core.somaxconn = 100000
net.ipv4.tcp_max_syn_backlog = 100000
net.core.netdev_max_backlog = 100000
net.core.rmem_max = 25165824
net.core.rmem_default = 25165824
net.core.wmem_max = 25165824
net.core.wmem_default = 65536
net.core.optmem_max = 25165824