Discussion:
throughput limit with ocproxy
Tom Rodriguez
2018-11-09 21:50:43 UTC
Permalink
I've been happily using openconnect with ocproxy but a while back I
upgraded my link and I realized that each connection is topping out at
about 1.5MB/s. This is measured with wget. If I use openconnect
directly then I get the full bandwidth as I'd expect. The other thing I
noticed is that if I run multiple downloads then I can actually saturate
the link but the individual downloads always top out at 1.5MB/s. This
is true for both mac and linux which sort of surprises me. Any
suggestions for a cause or how to investigate further? Thanks!

tom
Kevin Cernekee
2018-11-13 16:22:10 UTC
Permalink
Post by Tom Rodriguez
I've been happily using openconnect with ocproxy but a while back I
upgraded my link and I realized that each connection is topping out at
about 1.5MB/s. This is measured with wget. If I use openconnect
directly then I get the full bandwidth as I'd expect. The other thing I
noticed is that if I run multiple downloads then I can actually saturate
the link but the individual downloads always top out at 1.5MB/s. This
is true for both mac and linux which sort of surprises me. Any
suggestions for a cause or how to investigate further? Thanks!
On Linux I've mostly been using vpnns (from the ocproxy package) since
that lets the kernel handle TCP/IP. Does that help or is the
bottleneck elsewhere?
Tom Rodriguez
2018-11-13 21:56:41 UTC
Permalink
Post by Kevin Cernekee
Post by Tom Rodriguez
I've been happily using openconnect with ocproxy but a while back I
upgraded my link and I realized that each connection is topping out at
about 1.5MB/s. This is measured with wget. If I use openconnect
directly then I get the full bandwidth as I'd expect. The other thing I
noticed is that if I run multiple downloads then I can actually saturate
the link but the individual downloads always top out at 1.5MB/s. This
is true for both mac and linux which sort of surprises me. Any
suggestions for a cause or how to investigate further? Thanks!
On Linux I've mostly been using vpnns (from the ocproxy package) since
that lets the kernel handle TCP/IP. Does that help or is the
bottleneck elsewhere?
Thanks, I wasn't aware of that command. That does indeed fix the
throughput problem on linux though I'll have to rework some things to
take advantage of it. Unfortunately it's not available for the Mac. I
can work with it though. Having a partial solution is better than none.

So that suggests that the bottleneck is really in ocproxy. Clearly
openconnect can communicate with the tunnel process efficiently enough
to support the throughput. I've investigated ocproxy before to see if
there was something wrong with the LWIP configuration that was limiting
throughput but didn't see anything. Anyway, thanks for the vpnns tip.

tom
David Woodhouse
2018-11-13 22:01:06 UTC
Permalink
Post by Tom Rodriguez
Thanks, I wasn't aware of that command. That does indeed fix the
throughput problem on linux though I'll have to rework some things to
take advantage of it. Unfortunately it's not available for the Mac. I
can work with it though. Having a partial solution is better than none.
So that suggests that the bottleneck is really in ocproxy. Clearly
openconnect can communicate with the tunnel process efficiently enough
to support the throughput. I've investigated ocproxy before to see if
there was something wrong with the LWIP configuration that was limiting
throughput but didn't see anything. Anyway, thanks for the vpnns tip.
Does LWIP support SACK yet? Are you seeing packet loss? Do you see SACK
being used when you use vpnns and the kernel's stack?

Or maybe it's just TCP congestion algorithms and window sizing...
Tom Rodriguez
2018-11-16 20:46:34 UTC
Permalink
Post by David Woodhouse
Post by Tom Rodriguez
Thanks, I wasn't aware of that command. That does indeed fix the
throughput problem on linux though I'll have to rework some things to
take advantage of it. Unfortunately it's not available for the Mac. I
can work with it though. Having a partial solution is better than none.
So that suggests that the bottleneck is really in ocproxy. Clearly
openconnect can communicate with the tunnel process efficiently enough
to support the throughput. I've investigated ocproxy before to see if
there was something wrong with the LWIP configuration that was limiting
throughput but didn't see anything. Anyway, thanks for the vpnns tip.
Does LWIP support SACK yet? Are you seeing packet loss? Do you see SACK
being used when you use vpnns and the kernel's stack?
ocproxy usesfff lwip 1.4.0 which doesn't support SACK, though the latest
version does. I ported it to use the latest lwip in hopes that would
solve my problem but it made no difference to the throughput. I did
look at the tcpdump style output ocproxy -T produces and there appear be
some problems with packet loss and retransmission but that only occurs
when I'm not getting full throughput. I've included a snippet of the
log at the end of this message. The normal pattern is 2 data packets
and 1 ack but sometimes it seem to miss every other packet for a while
and just keeps acking the last received packet until it eventually
resyncs. I think this explains the occasional very bad throughput I see
with ocproxy. The openconnect verbose output doesn't seem to indicate
that it's the one dropping the packets so they must be real drops I guess.

Anyway the -T output of my best throughput with ocproxy looks pretty
much the same as the tcpdump of the fast throughput with vpnns. The wnd
and message sizes are slightly different.
Post by David Woodhouse
Or maybe it's just TCP congestion algorithms and window sizing...
Yeah I'm will to accept that it might that lwip just doesn't cope that
well the stream it's seeing but I assume it can do better than 1.5MB/s.

tom

This is a ocproxy -T dump of a bad retransmission segment during a
transfer that was only getting about 700KB/s instead of the normal 1.5MB/s.

10.159.148.137.64242 > 10.213.24.141.80: ack 3592752147 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592752147:3592753395(1248)
ack 6734 wnd 30016
10.213.24.141.80 > 10.159.148.137.64242: . 3592753395:3592754643(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592754643 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592754643:3592755891(1248)
ack 6734 wnd 30016
10.213.24.141.80 > 10.159.148.137.64242: . 3592755891:3592757139(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592758387:3592759635(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592759635:3592760883(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592760883:3592762131(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592762131:3592763379(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592763379:3592764627(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592764627:3592765875(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592765875:3592767123(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592767123:3592768371(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592768371:3592769619(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592769619:3592770867(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592770867:3592772115(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592772115:3592773363(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592773363:3592774611(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592774611:3592775859(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592775859:3592777107(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592777107:3592778355(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592778355:3592779603(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592779603:3592780851(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592780851:3592782099(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592782099:3592783347(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592783347:3592784595(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592784595:3592785843(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592785843:3592787091(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592787091:3592788339(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592788339:3592789587(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592789587:3592790835(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592790835:3592792083(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592792083:3592793331(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592793331:3592794579(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592794579:3592795827(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592795827:3592797075(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592797075:3592798323(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592798323:3592799571(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592799571:3592800819(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592800819:3592802067(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592802067:3592803315(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592803315:3592804563(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592804563:3592805811(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592805811:3592807059(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592807059:3592808307(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592808307:3592809555(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592809555:3592810803(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592810803:3592812051(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592757139:3592758387(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592812051 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592812051:3592813299(1248)
ack 6734 wnd 30016
10.213.24.141.80 > 10.159.148.137.64242: . 3592813299:3592814547(1248)
ack 6734 wnd 30016
10.159.148.137.64242 > 10.213.24.141.80: ack 3592814547 wnd 65534
10.213.24.141.80 > 10.159.148.137.64242: . 3592814547:3592815795(1248)
ack 6734 wnd 30016
10.213.24.141.80 > 10.159.148.137.64242: . 3592815795:3592817043(1248)
ack 6734 wnd 30016

Loading...