Discussion:
Diagnosing error "SSL read error: The TLS connection was non-properly terminated"
John Hendy
2014-04-17 21:29:54 UTC
Permalink
Greetings,


I finally got openconnect to work with my company's Cisco VPN system
via some various help from the web and a co-worker on setting up a
csd-wrapper. However, I'm getting constant disconnection/reconnection
behaviors. Here's the output from my recent session:
- http://pastebin.com/wyHTzjwR

That error is generated every few seconds. One internal site seems to
go on operating reasonably fine (though very slow), while my company
mail client (browser-based) won't send any emails and requests
frequent re-authentication.

Here's the ~/.cisco/csd-wrapper.sh script used:

$ cat ~/.cisco/csd-wrapper.sh
#!/bin/bash

shift

URL=
TICKET=
STUB=
GROUP=
CERTHASH=
LANGSELEN=

while [ "$1" ]; do
echo "PARAM '$1' ==> $2"
if [ "$1" == "-ticket" ]; then shift; TICKET=$1; fi
if [ "$1" == "-stub" ]; then shift; STUB=$1; fi
if [ "$1" == "-group" ]; then shift; GROUP=$1; fi
if [ "$1" == "-certhash" ]; then shift; CERTHASH=$1; fi
if [ "$1" == "-url" ]; then shift; URL=$1; fi
if [ "$1" == "-langselen" ];then shift; LANGSELEN=$1; fi
shift
done

echo "URL: $URL"
echo "TICKET: $TICKET"
echo "STUB: $STUB"
echo "GROUP: $GROUP"
echo "CERTHASH: $CERTHASH"
echo "LANGSELEN: $LANGSELEN"

#ARGS="-log debug -ticket $TICKET -stub $STUB -group $GROUP -host $URL
-certhash $CERTHASH"
ARGS="-log error -ticket $TICKET -stub $STUB -group $GROUP -host $URL
-certhash $CERTHASH"

$HOME/.cisco/hostscan/bin/cstub $ARGS


Is this the case of a simple openconnect argument I'm not using/need
to specify or something else? Consider me completely ignorant with
respect to network/tunneling/etc., but I'm happy to collect any other
information suggested and post back. This is what seemed obvious to
start with, and I couldn't find any hits for the exact error I'm
getting. In fact, searching google for the exact phrase "SSL read
error: The TLS connection was non-properly terminated" only gets me
the pastebin I just posted.

Is this an error message specific to my company, or should these
messages be standard across all of them?


Many thanks,
John
Kevin Cernekee
2014-04-17 23:18:42 UTC
Permalink
Post by John Hendy
I finally got openconnect to work with my company's Cisco VPN system
via some various help from the web and a co-worker on setting up a
csd-wrapper. However, I'm getting constant disconnection/reconnection
- http://pastebin.com/wyHTzjwR
That error is generated every few seconds. One internal site seems to
go on operating reasonably fine (though very slow), while my company
mail client (browser-based) won't send any emails and requests
frequent re-authentication.
I would not expect the CSD wrapper to interfere with a connection that
has already been established. It should be a one-shot deal,
pre-logon.

Can you confirm that cstub isn't running in the background while the
connection is up?
Post by John Hendy
Is this the case of a simple openconnect argument I'm not using/need
to specify or something else? Consider me completely ignorant with
respect to network/tunneling/etc., but I'm happy to collect any other
information suggested and post back. This is what seemed obvious to
start with, and I couldn't find any hits for the exact error I'm
getting. In fact, searching google for the exact phrase "SSL read
error: The TLS connection was non-properly terminated" only gets me
the pastebin I just posted.
Is this an error message specific to my company, or should these
messages be standard across all of them?
The error corresponds to GNUTLS_E_PREMATURE_TERMINATION

I think this means that we were expecting to read a TLS record, but
the connection was unexpectedly closed. You could check this with
tcpdump/wireshark and see if there is a TCP RST originating from the
other side.

What versions of openconnect and GnuTLS are you running? Have you
tried upgrading?
John Hendy
2014-04-18 00:23:37 UTC
Permalink
Post by Kevin Cernekee
Post by John Hendy
I finally got openconnect to work with my company's Cisco VPN system
via some various help from the web and a co-worker on setting up a
csd-wrapper. However, I'm getting constant disconnection/reconnection
- http://pastebin.com/wyHTzjwR
That error is generated every few seconds. One internal site seems to
go on operating reasonably fine (though very slow), while my company
mail client (browser-based) won't send any emails and requests
frequent re-authentication.
I would not expect the CSD wrapper to interfere with a connection that
has already been established. It should be a one-shot deal,
pre-logon.
Can you confirm that cstub isn't running in the background while the
connection is up?
I'm vpn'd in right now, and did `ps ax | grep -i cstub` with no hits.
Just in case, though, I did `ps as | grep -i cs` and get a hit for
cscan:

8731 pts/0 S+ 0:00 sudo openconnect --csd-wrapper
/home/jwhendy/.cisco/csd-wrapper.sh --csd-user jwhendy gra.3m.com
8732 pts/0 S+ 0:00 openconnect --csd-wrapper
/home/jwhendy/.cisco/csd-wrapper.sh --csd-user jwhendy gra.3m.com
8733 pts/0 Z+ 0:00 [csd-wrapper.sh] <defunct>
8757 pts/0 S+ 0:00 /home/jwhendy/.cisco/hostscan/bin/cscan

It looks like what you thought: csd-wrapper gets run and then stops
(when I quit openconnect, that defunct entry goes away). I also
noticed that when re-checking after being vpn'd for ~10min (with
openconnect still going), the cscan entry wasn't there anymore,
either. I quit and restarted openconnect and it looks like it ran for
~1min.
Post by Kevin Cernekee
Post by John Hendy
Is this the case of a simple openconnect argument I'm not using/need
to specify or something else? Consider me completely ignorant with
respect to network/tunneling/etc., but I'm happy to collect any other
information suggested and post back. This is what seemed obvious to
start with, and I couldn't find any hits for the exact error I'm
getting. In fact, searching google for the exact phrase "SSL read
error: The TLS connection was non-properly terminated" only gets me
the pastebin I just posted.
Is this an error message specific to my company, or should these
messages be standard across all of them?
The error corresponds to GNUTLS_E_PREMATURE_TERMINATION
I think this means that we were expecting to read a TLS record, but
the connection was unexpectedly closed. You could check this with
tcpdump/wireshark and see if there is a TCP RST originating from the
other side.
I've got tcpdump running with `tcpdump -i wlan0`, but (of course), I
can't get the issue to replicate now. I was at a coffee shop earlier
and am now at home, but it's happened at home as well, so perhaps I
just need to wait and then post back. Not having done this before,
would I just copy the output of tcpdump here near the time the SSL
errors are occurring? Is there any sensitive information from that I
need to redact? Current output looks like this (sorry for
over-censoring...):

19:09:01.574979 IP xxx.com.https > bigBang.60453: Flags [P.], seq
xxx:xxx, ack xxx, win xxx, options [nop,nop,TS val xxx ecr xxx],
length 1429
19:09:01.575013 IP xxx.com.https > bigBang.60453: Flags [P.], seq
xxx:xxx, ack xxx, win xxx, options [nop,nop,TS val xxx ecr xxx],
length 229
19:09:01.575028 IP bigBang.60453 > xxx.com.https: Flags [.], ack xxx,
win xxx, options [nop,nop,TS val xxx ecr xxx], length 0
19:09:01.594260 IP xxx.com.https > bigBang.60453: Flags [.], ack xxx,
win xxx, options [nop,nop,TS val xxx ecr xxx], length 0
19:09:01.594293 IP bigBang.60453 > xxx.com.https: Flags [P.], seq
xxx:xxx, ack xxx, win xxx, options [nop,nop,TS val xxx ecr xxx],
length 170
19:09:01.620287 IP xxx.com.https > bigBang.60453: Flags [.], ack xxx,
win xxx, options [nop,nop,TS val xxx ecr xxx], length 0
19:09:01.691203 IP xxx.com.https > bigBang.60453: Flags [P.], seq
xxx:xxx, ack xxx, win xxx, options [nop,nop,TS val xxx ecr xxx],
length 101
19:09:01.730066 IP bigBang.60453 > xxx.com.https: Flags [.], ack xxx,
win xxx, options [nop,nop,TS val xxx ecr xxx], length 0
19:09:02.099072 IP bigBang.60453 > xxx.com.https: Flags [P.], seq
xxx:xxx, ack xxx, win xxx, options [nop,nop,TS val xxx ecr xxx],
length 309
19:09:02.128060 IP xxx.com.https > bigBang.60453: Flags [.], ack xxx,
win xxx, options [nop,nop,TS val xxx ecr xxx], length 0

I'll pass along this info for now, and will just let openconnect run
until if and when it starts reconnecting, posting back with tcpdump
output. I also plan to remove my current version and install the git
version from Arch's AUR:
- https://aur.archlinux.org/packages/openconnect-git/

I'll try and reproduce the issue with that version and also report
back on the results.
Post by Kevin Cernekee
What versions of openconnect and GnuTLS are you running? Have you
tried upgrading?
Garsh. I'm realizing that what I thought was a decent first email
lacked some critical information... sorry!

Arch Linux, x86_64

$ uname -a
Linux bigBang 3.14.1-1-ARCH #1 SMP PREEMPT Mon Apr 14 20:40:47 CEST
2014 x86_64 GNU/Linux

$ sudo pacman -Qi openconnect # Arch's versioning... which appears
different than the below
Name : openconnect
Version : 1:5.03-1
Description : Open client for Cisco AnyConnect VPN
Architecture : x86_64
URL : http://www.infradead.org/openconnect.html
Licenses : GPL
Groups : None
Provides : None
Depends On : libxml2 gnutls libproxy vpnc
Optional Deps : None
Required By : None
Optional For : None
Conflicts With : None
Replaces : None
Installed Size : 1157.00 KiB
Packager : Bartłomiej Piotrowski <***@archlinux.org>
Build Date : Wed 26 Feb 2014 12:34:44 AM CST
Install Date : Wed 26 Mar 2014 08:13:19 PM CDT
Install Reason : Explicitly installed
Install Script : No
Validated By : Signature

$ openconnect --version
OpenConnect version v5.03
Using GnuTLS. Features present: PKCS#11, DTLS

$ sudo pacman -Q | grep gnutls
gnutls 3.3.0-1


Many thanks for the quick reply,
John


P.S. Probably not the place to do this, but since the page welcomes
updates... the list of available packages page could be updated to
list Arch Linux.
- http://www.infradead.org/openconnect/packages.html

Something like (to steal from Fedora's entry):

Arch Linux
- Both openconnect and network-manager-openconnect packages are
included in the Arch Linux extra repository. Install with `# pacman -S
openconnect` or `# pacman -S network-manager-openconnect`. (Whether or
not they are truly up to date would be openconnect's call; the current
version is 5.03, but I see 5.99 is out. Then again, the package is
flagged out of date, so it should be updated soon:
https://www.archlinux.org/packages/?name=openconnect.)
Kevin Cernekee
2014-04-18 01:01:46 UTC
Permalink
Post by John Hendy
It looks like what you thought: csd-wrapper gets run and then stops
(when I quit openconnect, that defunct entry goes away). I also
noticed that when re-checking after being vpn'd for ~10min (with
openconnect still going), the cscan entry wasn't there anymore,
either. I quit and restarted openconnect and it looks like it ran for
~1min.
Well, just to rule it out, you can try the attached csd.sh (which
skips running the trojan). You'll need the curl utility installed to
POST the policy info to the gateway.
Post by John Hendy
$ sudo pacman -Qi openconnect # Arch's versioning... which appears
different than the below
Name : openconnect
Version : 1:5.03-1
Description : Open client for Cisco AnyConnect VPN
Architecture : x86_64
URL : http://www.infradead.org/openconnect.html
Licenses : GPL
This should probably say LGPLv2.1.
Post by John Hendy
$ openconnect --version
OpenConnect version v5.03
Using GnuTLS. Features present: PKCS#11, DTLS
$ sudo pacman -Q | grep gnutls
gnutls 3.3.0-1
When I saw your pastebin I wondered whether it was an old build that
didn't have DTLS compiled in. But that doesn't seem to be the case.
So your client supports DTLS but you're getting a TLS-only connection
for some reason.

On a public wifi network I would worry about packet loss / congestion,
and maybe timeouts on long lived TCP sessions. DTLS would help with
all of those.

Could you post the result from connecting with "openconnect -v" so we
can see if the gateway has DTLS disabled?
John Hendy
2014-04-18 01:46:05 UTC
Permalink
Post by Kevin Cernekee
Post by John Hendy
It looks like what you thought: csd-wrapper gets run and then stops
(when I quit openconnect, that defunct entry goes away). I also
noticed that when re-checking after being vpn'd for ~10min (with
openconnect still going), the cscan entry wasn't there anymore,
either. I quit and restarted openconnect and it looks like it ran for
~1min.
Well, just to rule it out, you can try the attached csd.sh (which
skips running the trojan). You'll need the curl utility installed to
POST the policy info to the gateway.
$ sudo pacman -Q | grep curl
curl 7.36.0-1

I can't connect with that script -- my credentials get denied and
there's a message to contact my company IT Help Desk. If I recall
correctly, I used to get that message when trying with the anyconnect
client if I hadn't started the /etc/rc.d/hostscan service.
Post by Kevin Cernekee
Post by John Hendy
$ sudo pacman -Qi openconnect # Arch's versioning... which appears
different than the below
Name : openconnect
Version : 1:5.03-1
Description : Open client for Cisco AnyConnect VPN
Architecture : x86_64
URL : http://www.infradead.org/openconnect.html
Licenses : GPL
This should probably say LGPLv2.1.
I couldn't figure out how to contact the maintainer or packager for
the package, so I just submitted a bug report with the correct
information to let them know:
- https://bugs.archlinux.org/task/39927
Post by Kevin Cernekee
Post by John Hendy
$ openconnect --version
OpenConnect version v5.03
Using GnuTLS. Features present: PKCS#11, DTLS
$ sudo pacman -Q | grep gnutls
gnutls 3.3.0-1
When I saw your pastebin I wondered whether it was an old build that
didn't have DTLS compiled in. But that doesn't seem to be the case.
So your client supports DTLS but you're getting a TLS-only connection
for some reason.
On a public wifi network I would worry about packet loss / congestion,
and maybe timeouts on long lived TCP sessions. DTLS would help with
all of those.
Could you post the result from connecting with "openconnect -v" so we
can see if the gateway has DTLS disabled?
Here's the verbose output using the csd-wrapper.sh I posted earlier:
- http://pastebin.com/5ZcNpUuj

I terminated it shortly after initiating, as the stuff at the very end
looked to just be repeating at rapid pace. I can re-run a failed
attempt with your csd.sh if that would be useful.


Best regards,
John
Kevin Cernekee
2014-04-18 02:08:54 UTC
Permalink
Post by John Hendy
$ sudo pacman -Q | grep curl
curl 7.36.0-1
I can't connect with that script -- my credentials get denied and
there's a message to contact my company IT Help Desk. If I recall
correctly, I used to get that message when trying with the anyconnect
client if I hadn't started the /etc/rc.d/hostscan service.
Hmm, OK, I'm probably missing some data from the request.
Post by John Hendy
Post by Kevin Cernekee
Could you post the result from connecting with "openconnect -v" so we
can see if the gateway has DTLS disabled?
- http://pastebin.com/5ZcNpUuj
If DTLS is enabled on the gateway you should see some X-DTLS fields, like this:

Got CONNECT response: HTTP/1.1 200 OK
X-CSTP-Version: 1
X-CSTP-Address: 192.168.6.14
X-CSTP-Netmask: 255.255.255.0
X-CSTP-Address: 2001:db8::2
X-CSTP-Netmask: 2001:db8::2/32
X-CSTP-Lease-Duration: 1209600
X-CSTP-Session-Timeout: none
X-CSTP-Idle-Timeout: 36000
X-CSTP-Disconnected-Timeout: 36000
X-CSTP-Keep: true
X-CSTP-Tunnel-All-DNS: false
X-CSTP-Rekey-Time: 240
X-CSTP-Rekey-Method: new-tunnel
X-CSTP-DPD: 30
X-CSTP-Keepalive: 20
X-CSTP-MSIE-Proxy-Lockdown: true
X-CSTP-Smartcard-Removal-Disconnect: true
X-DTLS-Session-ID:
88697A32A530784A738CB60D4B715D9DEC9C9EF6274AB2D2A857BB80C2BCF52E
X-DTLS-Port: 443
X-DTLS-Keepalive: 20
X-DTLS-DPD: 30
X-DTLS-Rekey-Time: 240
X-CSTP-MTU: 1406
X-DTLS-CipherSuite: AES128-SHA
X-CSTP-Routing-Filtering-Ignore: false
X-CSTP-Quarantine: false
X-CSTP-Disable-Always-On-VPN: false
X-CSTP-TCP-Keepalive: true
X-CSTP-Post-Auth-XML: <elided>
CSTP connected. DPD 30, Keepalive 20
DTLS option X-DTLS-Session-ID :
88697A32A530784A738CB60D4B715D9DEC9C9EF6274AB2D2A857BB80C2BCF52E
DTLS option X-DTLS-Port : 443
DTLS option X-DTLS-Keepalive : 20
DTLS option X-DTLS-DPD : 30
DTLS option X-DTLS-Rekey-Time : 240
DTLS option X-DTLS-CipherSuite : AES128-SHA
DTLS initialised. DPD 30, Keepalive 20
Connected (script) as 192.168.6.14 + 2001:db8::2/32, using SSL
No work to do; sleeping for 20000 ms...
No work to do; sleeping for 20000 ms...
Established DTLS connection (using OpenSSL). Ciphersuite AES128-SHA.


If you can get in touch with your ASA admin, they can re-enable DTLS
(i.e. disable no-tls mode) with these commands:

config term
webvpn
enable outside

That is the first thing I would try if experiencing performance or
stability problems on a poor connection.
John Hendy
2014-04-18 03:21:49 UTC
Permalink
Post by Kevin Cernekee
Post by John Hendy
$ sudo pacman -Q | grep curl
curl 7.36.0-1
I can't connect with that script -- my credentials get denied and
there's a message to contact my company IT Help Desk. If I recall
correctly, I used to get that message when trying with the anyconnect
client if I hadn't started the /etc/rc.d/hostscan service.
Hmm, OK, I'm probably missing some data from the request.
Post by John Hendy
Post by Kevin Cernekee
Could you post the result from connecting with "openconnect -v" so we
can see if the gateway has DTLS disabled?
- http://pastebin.com/5ZcNpUuj
Got CONNECT response: HTTP/1.1 200 OK
X-CSTP-Version: 1
X-CSTP-Address: 192.168.6.14
X-CSTP-Netmask: 255.255.255.0
X-CSTP-Address: 2001:db8::2
X-CSTP-Netmask: 2001:db8::2/32
X-CSTP-Lease-Duration: 1209600
X-CSTP-Session-Timeout: none
X-CSTP-Idle-Timeout: 36000
X-CSTP-Disconnected-Timeout: 36000
X-CSTP-Keep: true
X-CSTP-Tunnel-All-DNS: false
X-CSTP-Rekey-Time: 240
X-CSTP-Rekey-Method: new-tunnel
X-CSTP-DPD: 30
X-CSTP-Keepalive: 20
X-CSTP-MSIE-Proxy-Lockdown: true
X-CSTP-Smartcard-Removal-Disconnect: true
88697A32A530784A738CB60D4B715D9DEC9C9EF6274AB2D2A857BB80C2BCF52E
X-DTLS-Port: 443
X-DTLS-Keepalive: 20
X-DTLS-DPD: 30
X-DTLS-Rekey-Time: 240
X-CSTP-MTU: 1406
X-DTLS-CipherSuite: AES128-SHA
X-CSTP-Routing-Filtering-Ignore: false
X-CSTP-Quarantine: false
X-CSTP-Disable-Always-On-VPN: false
X-CSTP-TCP-Keepalive: true
X-CSTP-Post-Auth-XML: <elided>
CSTP connected. DPD 30, Keepalive 20
88697A32A530784A738CB60D4B715D9DEC9C9EF6274AB2D2A857BB80C2BCF52E
DTLS option X-DTLS-Port : 443
DTLS option X-DTLS-Keepalive : 20
DTLS option X-DTLS-DPD : 30
DTLS option X-DTLS-Rekey-Time : 240
DTLS option X-DTLS-CipherSuite : AES128-SHA
DTLS initialised. DPD 30, Keepalive 20
Connected (script) as 192.168.6.14 + 2001:db8::2/32, using SSL
No work to do; sleeping for 20000 ms...
No work to do; sleeping for 20000 ms...
Established DTLS connection (using OpenSSL). Ciphersuite AES128-SHA.
If you can get in touch with your ASA admin, they can re-enable DTLS
config term
webvpn
enable outside
That is the first thing I would try if experiencing performance or
stability problems on a poor connection.
Hmmm. I can try. This is an 80k employee, world wide company, and I've
experienced approximately no response for other requests...

I'll start with an IT contact I have to test the waters :)

Thanks for the suggestion.
John
David Woodhouse
2014-04-18 19:16:38 UTC
Permalink
Post by Kevin Cernekee
The error corresponds to GNUTLS_E_PREMATURE_TERMINATION
I think this means that we were expecting to read a TLS record, but
the connection was unexpectedly closed. You could check this with
tcpdump/wireshark and see if there is a TCP RST originating from the
other side.
See commit 0a0fce467 ?
--
dwmw2
Kevin Cernekee
2014-04-18 19:31:56 UTC
Permalink
Post by David Woodhouse
Post by Kevin Cernekee
The error corresponds to GNUTLS_E_PREMATURE_TERMINATION
I think this means that we were expecting to read a TLS record, but
the connection was unexpectedly closed. You could check this with
tcpdump/wireshark and see if there is a TCP RST originating from the
other side.
See commit 0a0fce467 ?
Same error code, but in this case it's affecting the CSTP connection
rather than the initial HTTP handshaking:

http://pastebin.com/wyHTzjwR

I don't think we need a special check here because it automatically reconnects.
Loading...