kubectl port forwarding timeout issue
KubernetesKubectlKubernetes Problem Overview
While using kubectl port-forward function I was able to succeed in port forwarding a local port to a remote port. However it seems that after a few minutes idling the connection is dropped. Not sure why that is so.
Here is the command used to portforward:
kubectl --namespace somenamespace port-forward somepodname 50051:50051
Error message:
Forwarding from 127.0.0.1:50051 -> 50051
Forwarding from [::1]:50051 -> 50051
E1125 17:18:55.723715 9940 portforward.go:178] lost connection to pod
Was hoping to be able to keep the connection up
Kubernetes Solutions
Solution 1 - Kubernetes
Setting kube's streaming-connection-idle-timeout
to 0 should be a right solution, but if you don't want to change anything, you can use while-do construction
Format: while true; do <<YOUR COMMAND HERE>>; done
So just inputing in CLI: while true; do kubectl --namespace somenamespace port-forward somepodname 50051:50051; done
should keep kubectl reconnecting on connection lost
Solution 2 - Kubernetes
Seems there is a 5 minute timeout that can be overridden with kubelet
parameters:
https://github.com/kubernetes/kubernetes/issues/19231
>If you want to pass something higher than 5 minutes (or unlimited) into your kubelets, you can specify the streaming-connection-idle-timeout
. E.g.
--streaming-connection-idle-timeout=4h
to set it to 4 hours. Or:
--streaming-connection-idle-timeout=0
to make it unlimited.
(DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Solution 3 - Kubernetes
I solved this by keeping the connection alive, e.g. using curl or nc.
Forward the port:
kubectl --namespace somenamespace port-forward somepodname 50051:50051
In another terminal, keep the connection alive by reaching out to the port every 10 seconds:
while true ; do nc -vz 127.0.0.1 50051 ; sleep 10 ; done
Solution 4 - Kubernetes
For windows make such bat (God forgive me)
:1
oc port-forward PODNAME 8003:8080
goto 1
Solution 5 - Kubernetes
If you are running your Kubernetes cluster behind a load balancer (like HAProxy), it could happen that the timeout configured in kubelet is bigger than the timeout configured in the HAProxy.
For instance, the streamingConnectionIdleTimeout
setting in Kubelet by default is 4h:
$ kubectl proxy --port=8001 &
$ NODE_NAME="XXXX"; curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' | grep streaming
"streamingConnectionIdleTimeout": "4h0m0s",
But if in HAProxy (or your preferred LB) you have these settings:
defaults
timeout client 1m
timeout server 1m
...
Trying to execute a port-forwarding will timeout if you don't have any activity over the app:
$ date; kubectl port-forward service/XXXX 1234:80
Mon Jul 5 10:58:20 CEST 2021
Forwarding ...
# after a minute
E0705 10:59:21.217577 64160 portforward.go:233] lost connection to pod
In order to fix this, a solution would be to increase the timeout (be careful with this, because depending on your cluster it can have undesirable effects) or bypass the LB when doing port-forwarding connecting directly to the API server (if your environment allows it).