SSH Keys 101

Published on March 6, 2013.

I began by following GitHub’s basic instructions on how to generate a key pair and add it to my account, but I really didn’t have any clue what I was doing. It just worked, so I left it alone.

Since then, I’ve had to set up SSH key authentication on several local machines and remote servers, and things have become a mess. It’s time for some spring cleaning.

What is an SSH key pair?

SSH keys are a means of identifying yourself to an SSH server. They are useful because they allow you access to a server, via SSH, without having to manually enter your password each time.

SSH keys always come in pairs: one private key and one public key. The private key should be safely guarded, while the public key can be shared freely on any SSH server to which you would like to connect.

How does it work?

When you use the ssh command to connect to a server that has your public SSH key on file, a few things happen behind the scenes:

  1. The SSH server uses your public SSH key to create and send your machine a “challenge”. This challenge is basically a coded message that must be met with an appropriate response or else your connection attempt will be refused.
  2. The only thing that can decode this challenge and send the correct response is your corresponding private SSH key in the key pair. This encrypted private key usually lives in the ~/.ssh/ directory on your machine.
  3. If a correct response is sent back to the SSH server, your connection will be authenticated and you will receive access.

Managing SSH key pairs

Up until now, I’ve been using keys generated on my local machines, passphrase-less keys generated on remote servers, and whatever else seemed to work. This isn’t terribly secure and it has become unmanageable — confusing, even.

I came across a very helpful ServerFault thread and decided that having one key pair per local machine was the best option for me.

This has several implications.

  1. You will have to delete all existing SSH stuff and start fresh. Well, maybe not, but it helped me since I’m new to this stuff.
  2. You will have to generate one key pair on each of your workstations, and then authorize each public key on every server that you’d like to access.
  3. You will have to set up SSH agent forwarding so that you can use your local keys instead of leaving passphrase-less keys on every server.

At the moment, I only have two machines: a 15” MacBook Pro at home, and a 27” iMac at the office. This should be a piece of cake, right?

Welp. Not really.

Backing up and removing all SSH stuff

Since my situation was such a mess, I wanted to start over with some fresh keys. You don’t necessarily have to, but it relieved some of my confusion by starting from nothing.

This was not as easy as I had hoped, but going through the motions helped me learn some new things.

So, before you go on a deleting frenzy, make sure that PasswordAuthentication is set to “Yes” on your remote server(s), otherwise you will lock yourself out after deleting keys. And obviously, you will need to know your user passwords so you can connect to your servers the ol’ fashioned way.

PasswordAuthentication is set in the /etc/ssh/sshd_config file on most of my servers. It may be the /etc/ssh/ssh_config file instead.

Shell in to each remote server, and edit this file:

sudo nano /etc/ssh/sshd_config

Edit the following lines and save:

PasswordAuthentication Yes

Restart SSH:

sudo service ssh restart

Next, you will want to remove any references to your old keys from the ~/.ssh/known_hosts and ~/.ssh/authorized_keys files, and then remove the keys themselves. You may need to do this on your remote server(s) and your local machines.

To remove the references, simply open up those files, delete the old entries, and save them.

sudo nano ~/.ssh/authorized_keys
sudo nano ~/.ssh/known_hosts

After deleting the references to the old keys, you will want to delete the keys themselves. If necessary, backup the old keys prior to removing them. Again, do this on your remote server(s) and local machines if you have keys in both places.

cp ~/.ssh/id_rsa ~/.ssh/id_rsa.bak
cp ~/.ssh/ ~/.ssh/
rm ~/.ssh/id_rsa ~/.ssh/

Repeat this process on all of your remote servers and local machines, so that you are back to square one. You should have to type your user password every time you try and SSH into any of your remote servers.

Removing keys from Github

If you are like me and have some remote servers that need to connect to Github, then you’ve probably added a few SSH keys to your account. These will also need to be deleted.

Simply login to your Github account and go to this settings page to delete them.

Generating new local SSH key pairs

At this point I was pretty terrified with what I did, but confident I could come out the other side a victorious noob.

Creating new keys is pretty simple. You will be generating a new key on each local machine that you expect to use SSH. For me, that means creating a pair on my laptop and a pair on my work computer.

You can generate either DSA or RSA keys. I typically generate RSA keys because they have no key length limits, so you can make them larger and therefore more secure.

To generate a pair of public and private SSH keys, simply run this command in your terminal:

ssh-keygen -t rsa -b 4096 -C "your comment"

the -t flag stands for type, the -b flag stands for the key length in bits, and the -C flag stands for the comment included in the pair.

Comments in your SSH public keys can help differentiate copies of the same public key. Each copy of a public key can have its own unique comment. I like to specify the local machine in the comment (e.g. “iMac”, “MacBook”, whatever).

After running the previous ssh-keygen command, you will be prompted for the desired name and location of your private key. To accept the defaults, press return for each prompt.

Next, you will be prompted for a passphrase. Make it strong, and store it somewhere so you don’t have to remember it. I use 1Password.

If you create a passphrase-less key, you will want to be aware of the risks. Without a passphrase, your private key will be stored on your machine in an unencrypted form. If some jerk gets their hands on your private key, they can assume your identity on any SSH server that is using the corresponding public key in the pair. You also must trust your root user (if it isn’t you), because they can bypass file permissions and will be able to access your private key file at any time.

If you have an existing private key on your local machine that you would like to keep the same but add/change/remove the passphrase, then run the following command:

ssh-keygen -f ~/.ssh/id_rsa -p

Authorizing public keys on your remote server(s)

Once your new keys are generated, you will want to authorize your public keys for each server that you plan on accessing. There are several ways you can do this, but I prefer to stay in the terminal for this process, using the secure copy scp command:

scp ~/.ssh/

This command copies your public key ( to your home directory on the server. The colon : at the end of the command is important — it stands for your home directory.

Now, you will want to connect to your server via SSH, concatenate your public key to the end of the ~/.ssh/authorized_keys file using the cat command, and change the permissions so that the file is only readable and writable by you:

cat ~/ >> ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

After adding your public key to ~/.ssh/authorized_keys, remove your key from your home directory:

rm ~/

Repeat this process on every remote server, for each of your public keys.

Once you are done authorizing your new keys, remember to set PasswordAuthentication back to “No” in /etc/ssh/sshd_config, and restart ssh on each remote server.

When you SSH back in, you will be prompted for your SSH passphrase (if you entered one while creating your key pair). On OSX, you can select “Remember password in my keychain” so that you don’t have to type in the passphrase every time you use ssh to access your remote server(s). Enter the password and continue, and you should be connected to your server.

Configuring SSH agent forwarding

(These forwarding instructions were mostly extracted from this Github article.)

After you authorize your new SSH key pairs, it’s time to set up agent forwarding so you can actually use the suckers.

This nifty little technique allows you to use SSH keys stored on your local machines instead of leaving passphrase-less keys sitting on your remote servers. ssh-agent runs in the background, and keeps your key stored in memory so you don’t have to enter your passphrase every time you need to use the key.

To start, you will want to create/modify the ~/.ssh/config file to allow forwarding to your remote servers.

sudo nano ~/.ssh/config

Add the following, replacing with your remote server’s domain name or IP address:

	ForwardAgent yes

You will want to add a host entry for each remote server you would like to access. It’s more secure this way than to use a wildcard like Host *.

Add your new local keys to Github

Now you will want to add your new keys in the same spot where you removed them earlier in this process. You should now be able to access Github from your remote server(s)!


At this point, you may be all set up. In my case, I was still having trouble, specifically with Github. I was getting this mysterious error:

fatal: The remote end hung up unexpectedly

If you see this, you should follow the “Troubleshooting” section of this Github article.

For me, I needed to add each of my private keys to ssh-agent and everything was working again.

Extra security (optional)

These extra tweaks are to protect yourself from yourself. You will want to run these commands on your remote server(s).

chmod 400 ~/.ssh/authorized_keys
sudo chattr +i ~/.ssh/authorized_keys
sudo chattr +i ~/.ssh

The first command makes the file read-only by you, so that it cannot be accidentally edited…by you. The second command prevents you, the user, from simply changing the permissions back. The third command prevents you from accidentally renaming the ~/.ssh/ folder.

This of course assumes that you will not need to change the file. If you end up having to change something in the future, then you will have to reverse this process in order to edit the file again.


The learning curve for this was much higher than I had expected, but I’m glad I muscled through it. I am still very new to all of this, so if there is something I am doing wrong or could be doing better please feel free to flame me in the comments.

Leave a Comment