Debugging SSH using Keys

Summary

If you running Secure Shell (ssh) between one or more different machines you can greatly simplify your life by distributing ssh public/private keys to enable password-less login.  In addition, you can make machines more secure by disabling password log in and allowing only ssh key based login.  Lastly, many systems on the internet require ssh public/private key pairs for identity management – most notably GitHub.  This article discusses the process for creating, configuring, deploying and debugging ssh keys.

Story

I have been working like crazy on Linux, but, my main desktop computer is a Mac.  Which means that I often find myself ssh logging into my Linux box over the local network.  To simplify this process I long ago put ssh keys on the Linux box so that I would not have to type the password over and over again.  But, a day or two ago it stopped working and rather than fix it, I just retyped the password over and over and over and ….  Finally, this got annoying enough to fix, which lead me into the rabbit hole of debugging.  And, after figuring it out, I thought that I would write down what I did and how it works.

In this article when I say “ssh” it means the client side program which attaches to the server.  When I say “sshd” I mean the server side daemon, to which the client side “ssh” attaches.

How Does it Work?

SSH depends on a shared random secret key, also known as a symmetric key, to encrypt data.  In other words, BOTH sides of the connection need to known the same random key.  But, how do you get this done securely?  You either use a password (not the subject of this article – and only semi-secure) or you use a Public/Private key pair (the subject of this article).

In the big picture, when using the public/private key method, the client and the server do two things:

  1. Identify each other as valid using the public key of the other side i.e.  the server verifies the clients public key… and the client verifies servers public key
  2. The server and client exchange/create a random shared secret key by encrypting using the others public keys

Like seeming all Unix things, there are a mind blowing number of options and variations, but this is the basic idea.  I am not going to go into the specific details of what, when, how the public/private key encryption is doing, but there are plenty of sources in the internet which discuss it.

Here is the minimum that you need to know about the specifics for the public/private key method to work:

  • Both the server and client need to have their own public/private key pair
  • The server public key must exist in the client’s “~/.ssh/known_hosts” file
  • The clients public key must exist in the server’s “~user/.ssh/authorized_keys” file
  • The client must have access to the client private key (which could be an issue as the private key can be encrypted by another password)

Client SSH Files

These files exist on the ssh client and control how the client side of the system works.  They must have something in them that makes sense (more on this) and they must have the correct Unix permissions.

Path Permission Comment
~ o-w Your home directory cannot have write permissions for "other" or the sshd will not allow key based login … and you are also asking to get really hosed
~/.ssh 700 This directory contains all of the user client ssh configuration files and keys.
~/.ssh/id_rsa 600 The Private Key which is paired with the public key "id_rsa.pub".  This is the default file name for the private key, however, like everything in Unix, can be configured to be something else.  You really really really don't want this file shared with anyone.
~/.ssh/id_rsa.pub o-w The Public Key that is paired with the private key "id_rsa"
~/.ssh/config o-w The ssh client configuration options file.  This file will be read first by the ssh program, followed by the global configuration file.  This file is not required to exist if you want to use the default configuration (but there are some helpful things that you can put in this file).
/etc/ssh/ssh_config o-w The global configuration file for the ssh client.  This file is read after the users local configuration file.
~/.ssh/known_hosts o-w This file contains the public keys of known ssh servers.  This is the principal mechanism to prevent man-in-the-middle attacks on ssh.

Server SSHD Files

These files exist on the sshd server and control how the server side of the system works.  They must have something in them that makes sense (more on this) and they must have the correct unix permissions.  The “user” means the user you are trying to log in as on the server side of the connection (which does not have to be the same user as you are logged in as on the client)

Path Permission Comment
~user o-w The home directory of the "user" cannot have write permissions for "other" or the sshd will not allow key based login … in addition you are also asking to get really hosed
~user/.ssh 700 The server "user" ssh/sshd files directory
~user/.ssh/authorized_keys o-w This file contains all of the valid PUBLIC keys of the client ssh's which are allowed to log in as this user, probably id_rsa.pub (but it can be anything).  There can be multiple public keys concatenated onto the end of this file.
/etc/ssh/ o-w This directory contains the global sshd configuration files and key pairs used by the sshd server.  It also contains the ssh client configuration files which are used only by ssh clients initiated on this machine – don't be confused, as this machine can be both a server and a client.
/etc/ssh/sshd_config o-w This file contains the configuration information for the secure shell daemon.

Generating Keys using ssh-keygen

As I stated earlier you need to have a public/private key pair in order to use ssh in this configuration.  In order to create a pair of keys you use the OpenSSH program called ssh-keygen.

The program ssh-keygen will enable you to create SSH keys.  As stated earlier, ssh keys are the foundation of public key cryptography and come in a pair which are mathematically related.  The ssh-keygen program has about 5000 options, but for this purpose you only need 3 or even 2 or maybe even 1

  • “-t rsa” specifies that you want an RSA key (as opposed to a DSA, ECDSA, or ED25519) which might also work?
  • “-b 4096” makes a really long 4096-bit key instead of the default 2048-bit key (which also probably works)
  • “-C some comment” puts a comment in the key file (which you dont actually need but helps identify the key)

There is one more trick.  When you run the ssh-keygen program it will give you the option of a passphrase to encrypt your private key.  This renders your private key unusable unless you provide the passphrase, which is both good and bad, more on this later.

Here is and example:

  • ssh-keygen -t rsa -b 4096 -C “my_test_key”

When you run this command it will do something like this: (note to start out I will NOT enter a passphrase)

These key files are stored in OpenSSH format.  Here is an example of a the private key stored in the file “id_rsa”

And here is the public key stored in the file id_rsa.pub

Not to state the obvious, but if you loose control of the private key you also lose control of what you are protecting.  For example, right this minute, there are many many public/private keys that are checked into public GitHub.  Scanning for these keys is a very well known method for hackers to get you.

Notice that by default the ssh-keygen puts your new keys in ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub.  Which is good, as that is also where the ssh program expects to find them by default.

Configure the SSHD Server

Now that you have all of the pieces required to make the client side of an ssh connection work, let’s configure the server side.  For the server side to function correctly you need to have your client public key concatenated onto the end of the server file “~user/.ssh/authorized_keys” file.  In other words:

  1. Copy the id_rsa.pub onto the sshd server machine
  2. Log into sshd server machine
  3. cat id_rsa.pub >> ~user/.ssh/authorized_keys

Here is an example of an authorized key file.

The easiest way to accomplish this task is to use the OpenSSH program “ssh-copy-id”.  This program knows how to “do the needful” by adding your client public key to the correct file (authorized_keys) on the server machine.

Here is an example of running ssh-copy-id on the client machine to setup my ssh server machine linux.local for the user “arh”

  • On line 1 I run the ssh-copy-id (on the client) to copy my id_rsa.pub file to the server user “arh” on the machine “linux.local” (the server)
  • On line 5 it asks me if I am sure that I trust the linux.local machine (more on this later)
  • On line 8 ssh asks me the password for the user arh on the machine linux.local (see it doesn’t know me yet)
  • On line 10 it says that it added my key to the authorized_key file on the server linux.local
  • On line 12 it suggests how I can test that the process worked by logging into the linux.local machine
  • On line 15 I issue ssh arh@linux.local (on the client) to log into the remote ssh server machine (see Im a compliant person)
  • Then you can see that I am logged in without a password
  • Finally I look at the authorized_key  file (on the server) to show my newly install public key

The “ssh” known_hosts file

When I log into a remote machine and ACCEPT the public key from that machine, the ssh program will add the servers public key to the file ~/.ssh/known_hosts on the ssh client side.  Once the public key from the other side is added to your client known_hosts file, you will no longer be asked to verify the remote side.  This is great right?  Well, there is one rather serious catch here.  If you accept a public key which you don’t actually know is the public key of the remote machine, you might find yourself in a man-in-the-middle (MIM) situation.  This would be bad as the MIM decrypts all of your traffic before sending it onto the machine you really mean to talk to. First of all I will admit that I have answered “yes” to the question “Are you sure you want to continue connecting (yes/no/[fingerprint])? yes” about a million times without every looking into it until I wrote this article.

So what is a person to do?

Remember the program “ssh-keygen” from earlier? That program has another option “-l” which will generate a secure hash of a key file that you can use to compare to make sure that you have the correct public key. But where does the server keep its public key?  By default, the answer is in /etc/ssh/ssh_host_ecda_key.pub.

Lets walk through the process.  Here is the message from when I accepted the public key of the remote server.

Now, let me log into the machine “linux.local” and find out the SHA of the public key.

Compare the SHAs.  Sure enough, what I accepted as the remote public key matches to what is on the remote server.  All good.  In fact, here is my known_hosts file which shows the public key of the server linux.local.

And I can run the “ssh-keygen -lf known_hosts” to show the SHA of the keys in the file… (notice they still match)

SSH-Agent Daemon

If you use keys with no passphrase, all is good.  But, if you put a passphrase on your key, things get a little bit more interesting.  Why would I put a passphrase on a private key?  Private keys often give you loads of power, so having a password on the file gives you another layer of security.

Let me show you.  Let’s start by making a new set of keys, this time Ill put in a passphrase.

Then I use ssh-copy-id to copy my keys to the machine linux.local

Now when I try to log in, the ssh program asks me the passphrase for my keys.  Once I give it the passphrase it will log me in without having to type in the password for the machine linux.local.  Unfortunately, when I log out and try again, the ssh program will ask me for the passphrase again.  This is not much of an improvement over just typing the password for the local machine over and over again.

The problem here is that ssh does not know the passphrase to decrypt the key file “id_rsa”.  So nothing works.  Now what?  Does that mean I just keep typing the passphrase for the key file?  No!  You may have never noticed that there is often a daemon running called “ssh-agent”.  Here it is on my Mac.  This daemon is responsible for temporarily storing decrypted private keys.

When the ssh program needs to use a key it will inquire of the ssh-agent daemon to see if it exists there.  Which it will not be unless you add it to the daemon using the command “ssh-add”.    If you add a key that is encrypted, the ssh-agent (actually ssh-add) will query you for the passphrase.  It will then store the decrypted key temporarily in RAM.  Now when I ssh into linux.local everything works.

You can ask the ssh-agent what keys it has available by running “ssh-add -l”.  Notice that it identifies the key using the SHA of the key.

Now that the decrypted private key is ssh-add(ed) to the ssh-agent I can log in as many times as I need and the ssh client will not ask me to decrypt the key.  The problem is now that the next time I reboot, the ssh-agent will restart and you will need to add the key back into the ssh-agent.  At least I only have to type the passphrase one time per log in.

MacOS Considerations – the KeyChain

There is a way to remember the passphrase, at least on MacOs (and probably on Linux I just don’t know what it is).  That method is to use the MacOS keychain.  To use it, add the option -K to the ssh-add.  Here it is:

You can see the key by running the program “KeyChain Access” and then clicking on Passwords and searching for ssh, you will see the password for the file id_rsa is part of the keychain

When you double click, you can change or look at the password.

You can also see that your key is inside of the ssh-agent.

The last step in making this work is adding the instructions to use the KeyChain and the Agent to the ssh config file (which is ~/.ssh/config) :

The SSH Config File

One more file in this whole chain is the ssh config file which will let you do host by host specific configurations. The config file lets  you set the configuration that you want to use for different “Host”s also known as ssh servers.  The file is setup as blocks of text that start with “Host hostname”.  Then a list of option/value pairs.  Here is an example of my config:

When the ssh starts, it will read the config file, then attempt match the name of the host you are trying to log into with a host in the host file.  When it finds a matching host, all of the following option/value pairs will be applied.  You are allowed to 1) specify Host with wildcard * or 2) with a name.  In addition you are allowed to have multiple “Host” sections that match.  In this case the first option found that matches is taken.  Here is a clip of the verbose output (-v) from ssh where you can see that ssh applied options for the “*” case and the “linux” case

One trick which I never realized until I worked on this article is that you can give the host any name you want, it doesn’t have to be a “real” name.  Then when you provide the “HostName” option you specify the real name of the host.  This lets me create custom aliases for hosts on my network (or actually any network)

Some of the interesting options which I often use are:

Option Comment
HostName Specify the actual hostname.  This allows you to have an alias for a host (that is not a DNS name) that will be mapped to an actual name.  In the case above, my Mac has no idea what computer "linux" is, but it maps "linux" to "linux.local" which is a valid name.  Which means I can run "ssh arh@linux" and will get the same as if I typed "ssh arh@linux.local"
User The name of the user you want to log in as on the remote system.  Allows you to not specify the user with user@ on the command line.  In the case above I can type "ssh linux" instead of "ssh arh@linux" and it will log in as "arh"
IdentityFile Specificy which key pair you want to use for this host (allows you to override the default "id_rsa") and have multiple key files
UseKeychain On a Mac, look for the password for encrypted key files in the MacOS keychain
AddKeysToAgent After a private key has been decrypted, it will be added automatically the ssh-agent for future reuse (to keep you from retyping the password)

Debug

Remember that all this started when I was not able to figure out why I could not log into the machine “linux.local”.  To figure this out I used the “-v” verbose option to spit out tons of debugging information.  You can see on line 65 where it sort of told me the answer.

So I went and looked at my directories and sure enough, my home directory was setup with permission for “others” to write. That is super bad.  Change the permissions and everything is good.

Resources

Here are a number of useful links where different aspects of this problem are discussed.

ssh man page 

ssh-agent man page

ssh_config

GitHub SSH Discussion 1

GitHub SSH Discussion 2

GitHub SSH Discussion 3

https://ssh.org

https://ssh.com