Building and configuring OpenSSH

Introduction

With the Core-SDI Advisory on the SSH1 CRC-32 compensation attack detector vulnerability and the associated exploit running around in Fall 2001, many people are rushing to upgrade their vulnerable secure shell servers. The exploit appears to be easy to use and quite effective, rooting boxes right and left.

We were among those rushing around upgrading systems, and it was quite a chore. So we've written down some rough instructions that report how we did it on our own systems. These are not intended to be tutorial, but they might give somebody a start. These instructions only apply to UNIX systems, and in particular we have installed on Red Hat Linux 6.2 and Solaris 2.7.

Quick Links

Locate the tarballs
Building everything
Creating OpenSSH public keys
Configuring incoming SSH public key access
Using Van Dyke SecureCRT with OpenSSH
Post-public key configuration
Copying the binaries elsewhere
Manual host key generation
References
Upgrading for the ZLIB problem
Privilege Separation
Portability notes

Locate the tarballs

There are three separate packages that have to be installed, and each is found in a different place thoughout the World Wide Web.

zlib - This is a great file compression library, and from the official home page the mirrors can be located. We used version 1.1.4 in tarball zlib-1.1.4.tar.gz. Note that there is a substantial security issue in version 1.1.3 - upgrade and rebuild your server if you've got the older one.

OpenSSL - This is a secure socket library, and it is easy to find at the OpenSSL source code specifically do not need the "engine" variant, which uses hardware that does the heavy-duty crypto. We've noted that this web site is often unavailable.

OpenSSH - This is the SSH client & server itself, and it's really easy to get the wrong version. We found ours at ftp://ftp3.usa.openbsd.org/pub/OpenBSD/OpenSSH/portable/, and it's important to always look in the portable/ subdirectory - otherwise you'll get a BSD only version. Our tarball was openssh-3.0p1.tar.gz.

We typically do our work in a /home/source area, so the unpacking will look like this:

#  cd /home/source 
#  gtar -xzvf zlib-1.1.4.tar.gz 
#  gtar -xzvf openssl-0.9.7a.tar.gz 
#  gtar -xzvf openssh-3.5p1.tar.gz

Building everything

The three packages have to be built in order, and we do them each in turn:

#  cd zlib-1.1.4 
#  ./configure 
#  make 
#  make install 

#  cd ../openssl-XXXXX 
#  ./config                        Note: not './Configure'
#  make 
#  make test 
#  make install 

#  cd ../openssh-XXXXX 
#  ./configure --with-ipv4-default --with-md5-passwords 
#  make 

#  no "make install" yet!

We chose the arguments to ./configure by feeling around, and we think this is the right set for our environment. These instructions have been working for us on a number of systems.

--with-ipv4-default appears to tell SSH to use only IPv4 -- "regular" internet addresses -- instead of the next-generation IPv6 addresses. We're not sure if this is required or not.

--with-md5-passwords enables the better password hashing as found in Linux, and it probably isn't required on other systems or if you only use RSA authentication.

We suspect that --with-pam might also be useful, but so far the system is working fine for us without it.

NOTE - for recent releases of OpenSSH, this is where the sshd user and group should be added to support privilege separation. See the end of this document for the details.

Install Everything - First Time

If this is a first-time installation of the Secure Shell on this system, it's safe to just do a make install in the OpsnSSH area and it will drop its files in the proper places.

But if there is a working SSH installation then it's wise to go find the key setup files and save them off to the side first. This installation also parks files in different target directories than the older one we used, so it's wise to poke around and save or delete the old stuff. We typically look in these directories.

/etc
/usr/local/bin
/usr/local/sbin

Pay particular note to the ssh_config and sshd_config files, which define important server and client options. Save these files to the side.

It's also wise to keep an alternate method of accessing the system, such as the console or telnet, until you're really sure that you have gotten SSH installed properly. It's a drag to find (for instance) that sshd doesn't start properly on a reboot when you have no keyboard or monitor attached to the server under the desk. Been there, done that.

Once these precautions have been taken, do a make install in the SSH source directory, and it will put everything where it belongs.

Post-Install Configuration

Once SSH is installed, there are still a few configuration steps required. We're really not terribly strong with this, and in any case cannot provide a tutorial for SSH here. But we will note a few things that ought to be attended to after upgrading secure shell.

Insure that sshd gets launched properly at system startup. In our installation, the daemon is located in /usr/local/sbin instead of /usr/local/bin, and the /etc/rc.d/init.d/sshd script has to be modified to reflect the proper location.
The file we use lives in /etc/rc.d/init.d/sshd, and it contains:
```
#
# secure shell daemon
#
# chkconfig: 2345 90 05
# description: secure shell daemon
#

SSHD=/usr/local/sbin/sshd

[ -x $SSHD ] || exit 0

case "$1" in
  start)        echo "Starting secure shell daemon"
                eval $SSHD
                ;;
esac
```
Once this file is in place (in /etc/rc.d/init.d/sshd), it has to be linked into the proper positions in the other runlevels so it actually: gets called. This is most easily done with the chkconfig program that manages the startup and shutdown locations in the various run levels.
```
# cd /etc/rc.d/init.d
# chmod +x sshd
# chkconfig --add sshd
```
If you prefer to do it yourself,
```
# cd /etc/rc.d
# chmod +x init.d/sshd
# ln -s ../init.d/sshd rc2.d/S90sshd
# ln -s ../init.d/sshd rc3.d/S90sshd
# ln -s ../init.d/sshd rc5.d/S90sshd
```
Important - do not forget the chmod step: the Secure Shell daemon won't start unless the init.d/sshd file is executable, and we've been burned by this more than once. It really stinks to be unable to get into your system remotely after a reboot because you forgot to chmod a file.
Make sure the sshd_config file is set correctly. In our installation we found that the file was installed to /usr/local/etc instead of /etc, and that none of our customizations in the old version migrated to the new. In particular, we're careful to turn off PasswordAuthentication, though we are not sure this is a great idea if this is the first time you're using Secure Shell.
But in any case, be sure to disable the SSH version 1 protocol by setting Protocol 2 in the file. SSH1 is not secure and can be the source of all kinds of shenanigans.

Setting up Public Key support

We are not excited about allowing PasswordAuthentication for our secure shell daemons, because this provides limited system protection. It's true that it does forestall network sniffing -- and this is no small matter -- but it does not stop the bad guy from running a password guesser over the secure channel. In our work as penetration testers, we've gotten into systems via secure shell this way: by guessing or finding a password.

Much better is to use Public Key encryption, which has the potential for substantially better security as well as more convenience. We've been surprised how much we like public key support in SSH - much more than we expected.

In PK, each user creates a pair of keys: one public and one private, and the two are mathematically related. Both are very long strings of binary numbers, and they bear no relation to anything like a "password" or anything else human readable. The public key can be given away, but the private key is so important that it is further protected by a "pass phrase". This pass phrase actually encrypts the private key, and you'll need to type this key every time you need to use it.

The pass phrase should be something easy to type but impossible to guess: you'll be typing it a lot. At first it seems annoying to type this same phrase over and over, but quickly found that typing one long phrase for everything postively beats the pants off of remember many individual passwords for all the sites we visit. This has turned out to be a big win.

Public key support revolves around the .ssh/ subdirectory in each user's UNIX/Linux home directory, and a couple of important files live here. These include (but not an exhaustive list):

authorized_keys - list of SSH1 public keys (incoming)
authorized_keys2 - list of SSH2 publc keys (incoming)
identity.pub - SSH1 private key (outgoing)
identity - SSH1 private key (outgoing)
id_rsa.pub - SSH2 RSA public key (outgoing)
id_rsa - SSH2 RSA private key (outgoing)
id_dsa.pub - SSH2 DSA public key (outgoing)
id_dsa - SSH2 DSA private key (outgoing)

IMPORTANT - Nothing will work right unless the ownership and permissions are set correctly. The directory and all files must be owned by the user in question, and must be unreadable by anybody but the user. The ssh server will simply ignore files that don't meet this stringent rule - it's a good security precaution. Just remember to chmod og= ~/.ssh ~/.ssh/* every now and then to be sure.

Creating OpenSSH public keys

The ssh-keygen program is used to create a public key pair suitable for OpenSSH client and server use (there are other mechanisms for creating keys, we'll touch on the m later). It's possible to create both "user" and "host" keys, but since the host keys are usually created during the software installation process, we'll only touch on the user key.

There are many options for creating public key pairs, but we only care about making SSH version 2 RSA keys. This command creates the key pair, prompts for a pass phrase to encrypt the private key, and stores the two files in your ~/.ssh directory:

$ ssh-keygen -b 1024 -t rsa

-b 1024 specifies the number of bits to encrypt the key, and the consensus is that 1024 is fine: larger values just slow things down without adding appreciably to security. -t rsa specifies to use RSA encryption to produce an SSH version 2 key (the default is to create a version one key - bad news). It prompts for a filename to store the keys in (usually OK to keep the default), then asks for a pass phrase twice.

Choose a good pass phrase! Your private key is not very private if the very secret id_rsa file can be compromised.

Running the command should look like this:

$ ssh-keygen -b 1024 -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/friedl/.ssh/id_rsa): RETURN
Enter passphrase (empty for no passphrase): passphrase + RETURN
Enter same passphrase again: passphrase + RETURN
Your identification has been saved in /home/steve/.ssh/id_rsa.
Your public key has been saved in /home/steve/.ssh/id_rsa.pub.
The key fingerprint is:
67:2c:06:ff:5a:cf:36:64:11:20:74:90:9f:97:14:8e steve@unixwiz.net

Now the files ~/.ssh/id_rsa.pub and ~/.ssh/id_rsa contain your public and private keys, respectively. We'll use them shortly. The public key file is one line of ASCII, and it looks like this (though it's actually all one line):

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAx9aYnr7XwNw0Ei3X2ngxRjGg
kQpK/EqpkveGCvvbMoH84zqu3Us8jSC0D392JZAEohGSoe0dWMBFm9Y41VGZ
YmncwkfTQPFH1P0IvDw49aTAa2RJNFyVQANZCbSocDeTT0QpusuUj/v8hj7+
PqsUUpUXVQSDIhXBkWV+bJawc1c= steve@unixwiz.net

Again, there is no need to keep a public key secret.

Configuring incoming SSH public key access

When the ssh server is considering an incoming connection request via the public key mechanisms, it consults the ~/.ssh/authorized_keys2 file (ssh version 2) or ~/.ssh/authorized_keys (ssh version 1). Though the OpenSSH server supports both version 1 and 2, version 1 has numerous security issues and should not be used. We won't consider ssh version 1 again.

To enable a remote user to the local OpenSSH server, simply edit the user's ~/.ssh/authorized_keys2 file and append the user's public key -- one line -- to the file and save it. Do make sure that the file is owned by the user and is not readable by anybody else, but there is no need to restart the OpenSSH server to make it take effect.

Then, from the remote system, connect with the ssh command and the name of the remote host.

$ ssh www.example.com
The authenticity of host 'www.example.com (10.1.1.1)' can't be established.
RSA key fingerprint is 50:29:ac:84:60:42:31:f7:e0:4c:bc:d2:ff:de:ba:89.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'www.example.com,10.1.1.1' (RSA) to the list of known hosts.
Enter passphrase for key '/home/steve/.ssh/id_rsa': passphrase + RETURN
$ uname -a
Linux www.example.com 2.2.12-20 #1 Mon Sep 27 10:25:54 EDT 1999 i586 unknown
$

The first time you contact any host, you'll be warned that your local SSH installation doesn't know about the remote. When you confirm that you want to connect to it with yes, the name and fingerprint of the remote host is added to the ~/.ssh/known_hosts file. Then you won't be prompted again for this host.

After confirming the host, you'll enter your passphrase to decrypt your private key. If done correctly, the SSH client and server exchange your key information, and if all is well you're granted access to the remote system.

There is no reason that a single public key can't be put in multiple user accounts on the remote system: it's possible to store your public key in both /home/steve/.ssh and /root/.ssh directories. So logged in as a regular user, we can login as root on the remote system:

$ ssh -l root www.example.com

Using Van Dyke SecureCRT with OpenSSH

We use the fantastic telnet/serial/SSH client SecureCRT (from Van Dyke Technologies) to connect from our Windows (9x/ME/NT/2000) boxes to OpenSSH, and setup is straightforward.

After SecureCRT is installed on the Windows box, a pair of public/private keys must be created. From the menu, select Options:Global Options, and from this dialog box select SSH2, then click Create Identity File. Key creation is done just one time after installation. NOTE: these instructions are for SecureCRT v4.x.

Click Next in the Key Generation Wizard, then select an RSA - not DSA - key and click Next. In the next dialog box, enter your pass phrase (twice), then a comment that appears in the file. By convention this is your email address, but it can be any descriptive text that helps you identify the key. Then click Next.

In the next dialog box, select 1024 bits for the key and click Next. Then you are invited to move the mouse around: this provides important random inputs that help generate a key pair that will be hard to crack. This takes perhaps 30 seconds, and when finished you are asked to click Next.

In the final dialog box, you're asked to save your key files somewhere. It's not as important exactly where they go as long as you remember where that place is. You will need the public key file shortly. For our example we are using C:\identity2.pub.

After the key has been generated, locate the public key file and get it over to the UNIX system. Since this file is small and ASCII, it's often easiest to open it in notepad, select the whole thing, and "cut" it into the Windows notepad. Then, logged into the remote server (perhaps with telnet or via ssh using a password), and pasting it into a file. For instance:

$ cat > /tmp/identity.pub
"paste" the file here
^D              <-- control-D
$

Remember that the entry is one line, so you might need to use your favorite editor and join them together if pasting created multiple lines.

Once the file is over on the UNIX system, it has to be convered from the SecureCRT format (some kind of standard) to that used by OpenSSH. The ssh-keygen command is able to do this conversion for you:

$ ssh-keygen -i -f /tmp/identity2.pub >> ~/.ssh/authorized_keys2

Now your SecureCRT public key is installed, so you should be able to ssh into your UNIX box from Windows.

Post-public key configuration

Once the public key mechanisms are known to be working correctly, including testing of proper restart after reboot, we recommend some additional steps.

Disable telnetd - even with TCP wrappers limiting the IP addresses of incoming connection, we believe that telnet is just too dangerous to leave running. So we disable telnetd in the /etc/inetd.conf file by commenting out this line
```
#telnet stream  tcp     nowait  root    /usr/sbin/tcpd  in.telnetd
```
and "kicking" inetd by sending a hangup signal to it:
```
# ps -fCinetd
UID        PID  PPID  C STIME TTY          TIME CMD
root       413     1  0 Feb01 ?        00:00:00 [inetd]

# kill -1 413
```
We believe that newer systems use the xinetd.conf file, but we don't have this on any of our systems: the idea ought to be similar.
Now telnet won't accept connections any longer. The brave and determined might actually remove the telnet daemon altogether. We've done this on most of our systems.
Disable PasswordAuthentication - in the OpenSSH configuration file (often /usr/local/etc/sshd_config), find the setting for PasswordAuthentication and change the value to no. This then permits only public key authentication and prevents "regular" passwords from working. We feel strongly that allowing people to guess passwords is a really bad idea, and by insisting on RSA keys, a whole raft of shenanigans can be avoided.

We're quite sure that other options in the config file are important, but we've not really looked at them in much detail. Shame on us.

Copying the binaries elsewhere

Once OpenSSH has been installed on a platform, it's reasonable to want to install it elsewhere without having to actually build it. So we've noted the files that are installed by the make install process so that they can be copied (with permissions!) to some other system. This listing is relative to /usr/local.

$ gtar -tzvf OpenSSH-3.1p1z.tgz
-rws--x--x root/root    211396 2002-03-13 01:32:57 bin/ssh
-rwxr-xr-x root/root     26520 2002-03-13 01:32:57 bin/scp
-rwxr-xr-x root/root     75564 2002-03-13 01:32:57 bin/ssh-add
-rwxr-xr-x root/root     42428 2002-03-13 01:32:57 bin/ssh-agent
-rwxr-xr-x root/root     79640 2002-03-13 01:32:57 bin/ssh-keygen
-rwxr-xr-x root/root    136784 2002-03-13 01:32:58 bin/ssh-keyscan
-rwxr-xr-x root/root     46216 2002-03-13 01:32:58 bin/sftp
lrwxrwxrwx root/root         0 2002-03-13 01:32:59 bin/slogin -> ssh
-rwxr-xr-x root/root    230416 2002-03-13 01:32:58 sbin/sshd
-rw-r--r-- root/root       182 2002-03-13 02:31:06 sbin/generate-ssh-keys
-rw-r--r-- root/root     46525 2002-03-13 01:32:58 man/man1/ssh.1
-rw-r--r-- root/root      3261 2002-03-13 01:32:59 man/man1/scp.1
-rw-r--r-- root/root      4896 2002-03-13 01:32:59 man/man1/ssh-add.1
-rw-r--r-- root/root      6315 2002-03-13 01:32:59 man/man1/ssh-agent.1
-rw-r--r-- root/root     10081 2002-03-13 01:32:59 man/man1/ssh-keygen.1
-rw-r--r-- root/root      3890 2002-03-13 01:32:59 man/man1/ssh-keyscan.1
-rw-r--r-- root/root      7129 2002-03-13 01:32:59 man/man1/sftp.1
lrwxrwxrwx root/root         0 2002-03-13 01:32:59 man/man1/slogin.1 -> ssh.1
-rw-r--r-- root/root     43258 2002-03-13 01:32:59 man/man8/sshd.8
-rw-r--r-- root/root      2087 2002-03-13 01:32:59 man/man8/sftp-server.8
-rwxr-xr-x root/root     23712 2002-03-13 01:32:58 libexec/sftp-server
-rw-r--r-- root/root       568 2002-03-13 01:32:56 share/Ssh.bin
-rw-r--r-- root/root     88039 2002-03-13 01:32:59 etc/moduli
-rw-r--r-- root/root      1144 2002-03-13 01:32:59 etc/ssh_config
-rw-r--r-- root/root      2523 2002-03-13 01:32:59 etc/sshd_config

The only nonstandard file is sbin/generate-ssh-keys, which is a small shell script we have created: it simply creates the three key pairs that are required. Run this program one time after installation to create these keys.

Host Key Generation

The SSH daemon requires its own set of host keys, and these are created as part of the installation process. But when the binaries are copied to another machine, the keys must be generated manually. As with all keys, these are created with the ssh-keygen program, and the three sets of keys are created with:

# ssh-keygen -t rsa1 -f /usr/local/etc/ssh_host_key -N ""
# ssh-keygen -t dsa -f /usr/local/etc/ssh_host_dsa_key -N ""
# ssh-keygen -t rsa -f /usr/local/etc/ssh_host_rsa_key -N ""

References

This entire Tech Tip is very lame in comparision with the excellent O'Reilly book SSH, The Secure Shell: The Definitive Guide, which covers everything in much more detail. Once you've gotten a handle on the basic configuration of OpenSSH, we recommend picking up this book to get to the finer points. We'll update our document as we pick them up ourselves.

Upgrading for the ZLIB problem

There has been a problem reported in the ZLIB compression library that has prompted version 1.1.4. We've installed this and found it easy to do.

Fetch the updated ZLIB tarball: zlib-1.1.4.tar.gz from the ZLIB home page.

Unpack, build, and install the source:

#  cd /home/source 
#  gtar -xzvf zlib-1.1.4.tar.gz 
#  cd zlib-1.1.4 
#  ./configure 
#  make install

Edit the OpenSSH version file. In the OpenSSH directory is a file version.h, and we want to remind ourselves that we've made this fix even though we haven't actually patched OpenSSH directly. So we tack a z to the end:
```
#define SSH_VERSION     "OpenSSH_3.0p1z"
```
Rebuild OpenSSH. We first do a "make clean" to force the changes to take effect:
```
#  vi version.h 
#  make clean 
#  make 
```
The resulting sshd file is left in the current directory.

Install sshd. Doing make install updates everything, and we're not sure this is really necessary. So we just manually copy the file to the final place. Then we look for the "parent" SSH daemon (the one with the PPID of 1) and send it the hangup signal. Then restart the daemon.

#  strip sshd 
#  cp sshd /usr/local/sbin 

#  ps -fCnamed 
UID        PID  PPID  C STIME TTY          TIME CMD
root      4509     1  0 07:46 ?        00:00:00 /usr/local/sbin/sshd
root      4510  4509  0 07:46 ?        00:00:00 /usr/local/sbin/sshd
root      4534  4509  0 07:49 ?        00:00:00 /usr/local/sbin/sshd

#  kill -1 4509
#  /usr/local/sbin/sshd

Portability Notes

We've managed to build OpenSSH on several other platforms with more or less the same instructions for Linux, but a few notes might be helpful here.

SCO UNIX 5.0.6 with SCO cc

This took a while to get around some build issues, but we ultimately got it working just fine.

zlib-1.1.4 built and installed without incident
openssl-0.9.7 required that we edit the Configure file and find the line for "sco-cc". Remove -lresolv from this line - we didn't find this file on our system, and our attempts to build BIND9 on this platform failed due to the lack of a "long long" integer type. Then build, test, and install per instructions.
openssh-3.5p1 fails the "configure" process because the default cc output is COFF, not ELF. So the command to set this up is ./configure --with-ipv4-default CC="cc -belf". Then build and install per the usual instructions.

SCO UNIX 5.0.5 with GNU cc

This built and installed with the above instructions excepting configuring OpenSSH itself (SCO doesn't use MD5 shadow passwords)

# cd openssh-3.2.3p1 (with gcc)
# ./configure --with-ipv4-default

We installed the startup file in /etc/rc2.d/S99sshd. This information built as of:

zlib-1.1.4
openssl-0.9.6d
openssh-3.2.3p1

The sshd privsep user directory must be created manually with all the right permissions lest it complain about the modes.

# mkdir -m 700 /var/empty
# chown sshd /var/empty
# chgrp sshd /var/empty

Sun SOLARIS 2.7

This likewise built with the same procedure with the exception of the the OpenSSH configuration itself:

# cd openssh-3.2.2p1
# ./configure --with-ipv4-default --with-pam

Built with:

zlib-1.1.4
openssl-0.9.6d
openssh-3.2.2p1

Privilege Separation

Starting with OpenSSH 3.3.2, the privilege separation feature was introduced. This is a security provision that goes a long way to forestall full system compromise even if a vulnerability is found. The details are covered elsewhere, and our favorite is here.

But before SSHD is installed, both a sshd user and group must be added, and it looks in the password and group files like this:

/etc/group sshd:*:27: /etc/passwd sshd:*:27:27:sshd privsep:/var/empty:/sbin/nologin

Under Red Hat Linux 7.2, these commands do the trick:

# groupadd sshd # useradd -M -g sshd -c 'sshd privsep' -d /var/empty -s /sbin/nologin sshd # passwd -l sshd

The -M parameter suppresses creation of the home directory, as this is done by the "make install" procedure for OpenSSH.

Note further that the UsePrivilegeSeparation yes option must be set in the sshd_config file - this may be the default for new installations, but upgrades don't usually touch current values. Make sure this option is set.

Feedback on this document is very much welcomed by .

Navigate: More Tech Tips