Can't ssh to one of the switches.

[nabz0r Veteran]

So I have this 3750 stack switch which uses telnet to login to and today I wanted to change it to use ssh, but I can't login.
It seems that the switch doesn't send matching ciphers though the ssh config on both switches are identical. @BudMan and @sc302 any ideas? I am out of clue to be honest, and don't know what to do next.

 

SSH output from my mac:

Unable to negotiate with 172.20.13.1 port 22: no matching cipher found. Their offer: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc

Switch log:

.Sep  3 13:20:52.805: %SSH-3-NO_MATCH: No matching cipher found: client chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com server aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc

 

Config on the switches:

line con 0
 logging synchronous
line vty 0 15
 exec-timeout 10 0
 logging synchronous
 login
 transport input ssh
ip domain-name xxxx.se
crypto key generate rsa general-keys modulus 4096

 

This is the debug output from ssh on the switch that doesn't work:

debug1: Local version string SSH-2.0-OpenSSH_7.8
debug1: Remote protocol version 2.0, remote software version Cisco-1.25
debug1: match: Cisco-1.25 pat Cisco-1.* compat 0x60000000
debug1: Authenticating to 172.20.13.1:22 as 'admin'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group14-sha1
debug1: kex: host key algorithm: ssh-rsa
Unable to negotiate with 172.20.13.1 port 22: no matching cipher found. Their offer: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc

c3750x#sh ip ssh 
SSH Enabled - version 2.0
Authentication timeout: 120 secs; Authentication retries: 3
Minimum expected Diffie Hellman key size : 4096 bits
IOS Keys in SECSH format(ssh-rsa, base64 encoded):
ssh-rsa <REMOVED>

Debug output from the switch that does work:

debug1: Local version string SSH-2.0-OpenSSH_7.8
debug1: Remote protocol version 2.0, remote software version Cisco-1.25
debug1: match: Cisco-1.25 pat Cisco-1.* compat 0x60000000
debug1: Authenticating to 172.30.49.3:22 as 'admin'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group14-sha1
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: none
debug1: sending SSH2_MSG_KEXDH_INIT
debug1: expecting SSH2_MSG_KEXDH_REPLY

stk7#sh ip ssh 
SSH Enabled - version 2.0
Authentication methods:publickey,keyboard-interactive,password
Encryption Algorithms:aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
MAC Algorithms:hmac-sha1,hmac-sha1-96
Authentication timeout: 120 secs; Authentication retries: 3
Minimum expected Diffie Hellman key size : 4096 bits
IOS Keys in SECSH format(ssh-rsa, base64 encoded):
ssh-rsa <REMOVED>

 

Edited September 3, 2018 by nabz0r

[+BudMan MVC]

What IOS are you switches running?  New versions of openssh are locked down to not accept OLD stuff... You will have to change your client to allow for old ciphers and hmac..  Cisco is a bit dated on keeping up with ciphers..  Run your client in -vvv when you connect to see the full list of what was offered and what was tried, etc.

[nabz0r Veteran]

Here is the complete output from my client. Btw, both are 3750x switches.

The IOS on the switch that doesn't work: 15.2(1)E1

On the one that works: 15.0(2)SE10a

 ⇝ mbp ⇜ ~/.ssh ssh -vvv XXX-3750
OpenSSH_7.8p1, OpenSSL 1.0.2p  14 Aug 2018
debug1: Reading configuration data /Users/walwar/.ssh/config
debug1: /Users/walwar/.ssh/config line 105: Applying options for XXX-3750
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname 172.20.13.1 is address
debug2: ssh_connect_direct
debug1: Connecting to 172.20.13.1 [172.20.13.1] port 22.
debug1: Connection established.
debug1: identity file /Users/walwar/.ssh/id_rsa type 0
debug1: identity file /Users/walwar/.ssh/id_rsa-cert type -1
debug1: identity file /Users/walwar/.ssh/id_dsa type -1
debug1: identity file /Users/walwar/.ssh/id_dsa-cert type -1
debug1: identity file /Users/walwar/.ssh/id_ecdsa type -1
debug1: identity file /Users/walwar/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/walwar/.ssh/id_ed25519 type -1
debug1: identity file /Users/walwar/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/walwar/.ssh/id_xmss type -1
debug1: identity file /Users/walwar/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.8
debug1: Remote protocol version 2.0, remote software version Cisco-1.25
debug1: match: Cisco-1.25 pat Cisco-1.* compat 0x60000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 172.20.13.1:22 as 'admin'
debug3: hostkeys_foreach: reading file "/Users/walwar/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /Users/walwar/.ssh/known_hosts:38
debug3: load_hostkeys: loaded 1 keys from 172.20.13.1
debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa
debug2: ciphers ctos: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96
debug2: MACs stoc: hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: diffie-hellman-group14-sha1
debug1: kex: host key algorithm: ssh-rsa
Unable to negotiate with 172.20.13.1 port 22: no matching cipher found. Their offer: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc

 

Edited September 4, 2018 by nabz0r

[nabz0r Veteran]

I guess I figured it out, it works when I do (below command) and obviously I don't want this workaround. Is it so that new version use aes256-ctr instead of aes256-crc because it's more "safe" and can reach higher speed.

 ⇝ mbp ⇜ ~ ssh  -c aes256-cbc admin@172.20.13.1
admin@172.20.13.1's password:

 

[GrayW]

A couple of options:

 

1. Edit your local .ssh/config file by removing the #'s in front of the lines of the offered cipher (as budman said)
2. Add specific host configs within your .ssh/config file to specify which cipher you want to use for which host (basically your -c without you having to remember to do it each time)- http://www.openssh.com/legacy.html

3. Depending on the number of switches in question, it could be easier to give in and use the Cisco CLI Analyzer tool (free but requires Cisco login) - https://www.cisco.com/c/en/us/support/web/tools-catalog.html

4. Spend time trying to get the switch to behave by regenerating the ssh configs with specific algorithms (probably the 'best practice' style answer)

 

It depends on your work environment I guess. I, personally don't have the time available to me to do 2 and 4 (despite them being the better options and starting this way) and with a sizeable estate of varying switch models and versions, opted for 3 which hasn't failed me so far.

 

Personally, I'm trying to push to a more unified IOS version across the estate with a git-style workflow for configuration changes...  But, as it generally takes 3 weeks for me to get 5 minutes of damn downtime to replace a cable half of the time, I'll continue dreaming!

 

There are probably other options available to you, but these are the ones I run through most often.

[nabz0r Veteran]

38 minutes ago, GrayW said:

A couple of options:

 

1. Edit your local .ssh/config file by removing the #'s in front of the lines of the offered cipher (as budman said)

1. I had already tried that but still couldn't ssh to the device.

2. Add specific host configs within your .ssh/config file to specify which cipher you want to use for which host (basically your -c without you having to remember to do it each time)- http://www.openssh.com/legacy.html

2. It's in my config file, with the option ciphers aes256-cbc but the problem is that I am not the only network engineer who access these devices so...
 

3. Depending on the number of switches in question, it could be easier to give in and use the Cisco CLI Analyzer tool (free but requires Cisco login) - https://www.cisco.com/c/en/us/support/web/tools-catalog.html

3. We already use SecureCRT and I've already tried to ssh from it with no luck.
 

4. Spend time trying to get the switch to behave by regenerating the ssh configs with specific algorithms (probably the 'best practice' style answer)

4. Not an option (and I don't like to have special configs for some of the switches)
 

It depends on your work environment I guess. I, personally don't have the time available to me to do 2 and 4 (despite them being the better options and starting this way) and with a sizeable estate of varying switch models and versions, opted for 3 which hasn't failed me so far.

 

Personally, I'm trying to push to a more unified IOS version across the estate with a git-style workflow for configuration changes...  But, as it generally takes 3 weeks for me to get 5 minutes of damn downtime to replace a cable half of the time, I'll continue dreaming!

Same here, we have almost unified IOS version on all the swtiches and routers, but on this one it's not possible as of  now, though I am going to upgrade and that only might solve the problem.
 

There are probably other options available to you, but these are the ones I run through most often.

 

[+BudMan MVC]

You just need to update your client to use the ciphers offered by default..

 

But you should really move to the current ios, and use of the current ciphers and algo's - old stuff is retired for a reason.. Since its not secure.. You should probe be using a chacha if your switches supported it.  But like I said cisco is always behind!!!

 

With securecrt, same thing.. you have to enable the ciphers you want to use.

 

You can do the same with openssh client... What versions of clients are you running openssh is 7.8p1 I believe and securecrt 8.5 released a while ago.. You need to make sure your using current clients so that they support current best use ciphers and algo's

 

Yeah 7.8p1 is current

C:\>ssh -V
OpenSSH_7.8p1, OpenSSL 1.0.2o  27 Mar 2018

 

You just need to make sure your client and servers can agree without having to call it out on the cmdline is all.

 

edit... What version of ssh client are you using.. You know the OLD key algo's have been deprecated... You might for example have to add

KexAlgorithms +diffie-hellman-group1-sha1

 

To your config file.. Again you need to just make sure your ssh client and server are using the same cipher, algo, mac -- their lists of offered and accept have to be able to agree.  Then you can just ssh in without having to do anything on cmd line.

 

edit: Yeah cisco sucks!  big time... I remove dh from my kex in securecrt and got this

key exchange failed.
No compatible key-exchange method. The server supports these methods: diffie-hellman-group,diffie-hellman

 

Will have to look if can update this on my switch.. Yup just did it on my 7.8p1 client and had to add the above config to enable the old stuff or get this.

 

debug1: kex: algorithm: (no match)
Unable to negotiate with 192.168.9.99 port 22: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1

 

edit:  Just noticed yeah your using 7.8 which is current and for sure is going to have the legacy stuff disabled out of the box... This url should be very helpful for you

https://www.openssh.com/legacy.html

 

 

 

 

[nabz0r Veteran]

I believe my client and my securecrt are up to date. To be honest, I don't have the energy to troubleshoot this anymore (I was kind of sure it is because that my Cisco switch using old algo and ciphers since it haven't been updated in 3 years, 39 weeks, 3 days, 15 hours, 35 minutes) I will update the switch in couple of weeks and hopefully that will solve the problem. I worked at customer site, the 6500 and 4500 haven't been rebooted in almot 10 years.

This site we just took over the operation and almost EVERYTHING in the network is out of date.. Let alone the ASRs are not in sync nor the HSRP configs are identical, the funny thins is that the responsible guy calls himself network architect. I just get so ###### when I see out of date stuff in the networks.

 

 ⇝ mbp ⇜ ~ ssh -V
OpenSSH_7.8p1, OpenSSL 1.0.2p  14 Aug 2018

 

Securecrt

[+BudMan MVC]

Yeah you are current   7.8 for sure has depreciated the older dh stuff..   So when talking to old ######, or even ###### that doesn't update their ssh stuff like cisco.. Lets get some chacha love around here Cisco.. And should be using ECDH for their key exchange..

 

When you get around to it.. Should prob take a look here

https://infosec.mozilla.org/guidelines/openssh

 

Some guidelines to work with get your sshd and clients more secure.. For example here is me connecting to one of my linux boxes via ssh

debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none