tcp_tw_reuse vs tcp_tw_recycle : Which to use (or both)?

I have a website and application which use a significant number of connections. It normally has about 3,000 connections statically open, and can receive anywhere from 5,000 to 50,000 connection attempts in a few seconds time frame.

I have had the problem of running out of local ports to open new connections due to TIME_WAIT status sockets. Even with tcp_fin_timeout set to a low value (1-5), this seemed to just be causing too much overhead/slowdown, and it would still occasionally be unable to open a new socket.

I've looked at tcp_tw_reuse and tcp_tw_recycle, but I am not sure which of these would be the preferred choice, or if using both of them is an option.

According to Linux documentation, you should use the TCP_TW_REUSE flag to allow reusing sockets in TIME_WAIT state for new connections.

It seems to be a good option when dealing with a web server that have to handle many short TCP connections left in a TIME_WAIT state.

As described here, The TCP_TW_RECYCLE could cause some problems when using load balancers...

EDIT (to add some warnings ;) ):

as mentionned in comment by @raittes, the "problems when using load balancers" is about public-facing servers. When recycle is enabled, the server can't distinguish new incoming connections from different clients behind the same NAT device.

NOTE: net.ipv4.tcp_tw_recycle has been removed from Linux in 4.12 (4396e46187ca tcp: remove tcp_tw_recycle).

SOURCE: https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux

pevik mentioned an interesting blog post going the extra mile in describing all available options at the time.

Modifying kernel options must be seen as a last-resort option, and shall generally be avoided unless you know what you are doing... if that were the case you would not be asking for help over here. Hence, I would advise against doing that.

The most suitable piece of advice I can provide is pointing out the part describing what a network connection is: quadruplets (client address, client port, server address, server port).

If you can make the available ports pool bigger, you will be able to accept more concurrent connections:

• Client address & client ports you cannot multiply (out of your control)
• Server ports: you can only change by tweaking a kernel parameter: less critical than changing TCP buckets or reuse, if you know how much ports you need to leave available for other processes on your system
• Server addresses: adding addresses to your host and balancing traffic on them:
  • behind L4 systems already sized for your load or directly
  • resolving your domain name to multiple IP addresses (and hoping the load will be shared across addresses through DNS for instance)