Systems exposed to the internet are heavily challenged to keep the bad guys out, and keeping up with the latest security patches is not always easy. So, the wise admin will attempt to institute systemic steps to limit the damage should a compromise occur, and one excellent method is the use of a chroot() jail.

This document touches on how chroot works and discusses some best practices that developers and administators can use to make their installations more secure.

Background on chroot

The hierarchy that forms the top of the jailed tree The chroot system call is found in all versions of UNIX that we know of, and it serves to create a temporary root directory for a running process, and it's a way of taking a limited hierarchy of a filesystem (say, /chroot/named) and making this /fill in here/

How to break out of jail

There are well-known techniques used to escape from jail, but the most common one requires root privileges inside the jail. The idea is for the program to do a chroot to a subdirectory, leaving the current directory outside the jail.

We'll add more notes on ways to break out of a jail - which is meant more to show what must be protected against than it is as a how-to for jailbreakers -- but we've found a good article on chroot in general here.

• Use mknod to create a raw disk device, thereby doing pretty much anything you like to the system.

• Use mknod to create /dev/mem and modify kernel memory

• Find a carelessly-left hard link that leads outside the jail (though symbolic links don't escape jail, hard links do).

• Use ptrace to trace a process living outside the jail. We may be able to modify this program to do our bad stuff on our behalf.

General chroot principles

Many of these points may end up being overly petty in practice, in that there are only so many layers of defense that a workable system can use, but we'll present all we can think of and let you pick and choose. An overriding principle is "What if the bad guy somehow does X? How can we limit our exposure".

Our general concern is mostly about remote buffer overflows, and this can give the bad guy complete control over our CPU: all our steps are designed to limit the damage should this unfortunate circumstance arise.

• Run in the jail as a non-root user - A chroot jail is not impervious to escape, but it not easy and requires root permission in the jail itself, so we must take steps to limit this possibility. By running the jail as a non-root user, it's as secure as we know how to make it. It may be necessary for the daemon to launch as root in order to do a few tasks that require these permissions (say, binding to a low-numbered port), but the program must "give up" its root permissions after doing so.

  We believe that this single factor is the most important one in setting up a jail properly.

• "Give up" permissions correctly - We've seen situations where on some operating systems, a program can jump back and forth between a non-root user and root by use of a "saved" uid, and this has been exploited by the bad guy who get root. We're a little shy to reveal that we really don't remember the details, but we're quite sure that it's an issue so we mention it here in the hopes that somebody will fill us in.

• Keep as little in the jail as possible, which limits what can be compromised should a vulnerability be discovered. Often this requires development support to do some "preloading" of non-jailed files before the chroot operation itself is performed (we'll touch on this a bit more later). But we're quite ruthless in removing things from the jail when possible.

• Limit non-jail running of jailed binaries - For systems that do not have a command-line option for running chroot, the only alternative is to create a wrapper program. This wrapper will perform the key chroot operation, give up root permission, and then execute the jailed binary.

  The wrapper must be run as root (only chroot can perform this operation), but the wrapper itself must not be found in the jail. Otherwise an intruder could quietly compromise the wrapper, and the next time the system is launched, the intruder's program would be run as root in a non-jailed environment. This is complete compromise.

• Have root own as many jailed files as possible - This limits the ability of the intruder to make changes should a compromise occur. Our feeling is that the most likely cause of penetration will be the buffer overflow exploit in which the intruder executes arbitrary code in environment, and for files that the jailed system need not ever write to, making them readonly and owned by root means that the penetration can't chmod the file before writing to it. This rule applies to directories as well.

• Drastically limit all permissions of files and directories. Our feeling is that if a permission bit is not required, it should not be set. For instance, the jailed "/dev/" directory should be of mode d--x--x--x with owner = root. Even though the only thing in the directory is /dev/null, forbidding searching of a directory strikes us as prodent practice across the board when it's known to work.

• Create a permissions-setting script -- when first setting up the jail, many of the permission-related knobs are tweaked by hand as we gradually tighten things up, looking for things to break (at which point the knob is eased back a bit). This research is intricate, and the knowledged gained really ought to be represented in source code somewhere.

  We typically create a small shell script -- living outside the jail -- that sets the owner, group, and permissions mode on every file in the jailed environment. It always starts with a few recursive change-everything options to hardcode everything to very tight permissions, then relaxes the settings on the files that can tolerate this. It's important to include documentation in the script on why particular permissions are relaxed, as well as describing why certain files are found in the jail in the first place.

  Once this script is created, we typically make all of our permissions-related changed here and then re-run the script to make them take effect. This is the only way that we can be sure that our script matches the running environment. A great side benefit of the permission script is that it serves as documentation to the next person setting up a similar environment.

  A sample permission script that we use for one of our projects (running BIND in a chroot jail). The specific details aren't really important, but this gives an idea

  
  cd /chroot/named
  
  # by default, root owns /everything/ and only root can write
  # but directories have to be executable too.
  
  chown -R root.named .
  
  find .         -print | xargs chmod u=rw,og=r      # *all* files
  find . -type d -print | xargs chmod u=rwx,og=rx    # directories
  
  # the "secondaries" directory is where we park files from
  # master nameservers, and named needs to be able to update
  # these files and create new ones.
  
  find conf/secondaries -type f -print | xargs chown named.named
  find conf/secondaries -type f -print | xargs chmod ug=r,o=
  chown root.named conf/secondaries
  chmod ug=rwx,o=   conf/secondaries
  
  # the var/run business is for the PID file
  chown root.root  var
  chmod u=rwx,og=x var
  
  chown root.named  var/run
  chmod ug=rwx,o=rx var/run

• Try to do the chroot operation inside the daemon itself rather than rely on the explicit chroot command (this requires source code modifications). A daemon that has its own internal chroot can often park the executable located outside the jail: this is a big win because an intruder is not able to ever infect the binary directly.

  But the more immediate benefit is that shared libraries and other startup files can be automatically loaded from the full system and need not be located inside the jail. This not only makes the system safer -- less exposure to the outside -- but also makes it easier to set up.

  In many cases, even configuration files can be loaded from outside the jail, though this won't usually work if the daemon includes any kind of "reread config files" option.

• Preload dynamically loaded objects - For developers adding chroot support to programs, consider operations that require access to full-system resources and perform them before closing the jail door. These steps are often not entirely obvious at first and require some trial and error, but we've found several that qualifye.

  Many systems load nameservice resolver clients dynamically at runtime, and they are not included in the shared objects bound to the executables. We have found that simply calling gethostbyname one time before the jail door is closed will load all the appropriate libraries required, so that later nameservice requests are handled properly:

  (void) gethostbyname("localhost");

  We believe that syslogging operations fall in this category too, as many systems uses UNIX domain sockets for this and require access to the socket that syslogd is listening on. We've not done the modifications required for syslog support and cannot offer any specific suggestions. We believe that Solaris -- with its use of "doors" -- is an added complication.

  For daemons that permit cmdline parameters to select the runtime users and group (after giving up root), the mapping of name to UID and GID must be done before the chroot operation so that the system-wide /etc/passwd and related files are used, not the one inside the jail. See the next section for the rationale.

  This bit of C code shows the idea of how the user lookup should be performed separately from the user ID changing:

  if ( geteuid() == 0 )
  {
  struct passwd *userent = 0;
  
      if ( (run_as_user != 0) && (userent = getpwnam(run_as_user)) == 0 )
      {
          /* ERROR */
      }
  
      chroot( working_dir );
  
      if ( userent )
          setuid(userent);
  
      ...
  
  

• Avoid using the jailed /etc/passwd file, particularly for name to UID mapping used to determine the runtime user ID of the daemon. The mapping involves scanning the passwd file for the given name (say, named) and finding the user ID associated with it. If the bad guy somehow manages to compromise the jailed passwd file, it's possible that the UID for the runtime user could be changed to zero, which is root. This will take effect the next time the daemon restarts.

  The bad guy shouldn't be able to compromise this file in the first place, because it should not be writable by the running user, but it's not out of the question that the daemon could somehow retain a writable file descriptor that the buffer overflow could use to modify the file: we believe we have seen this happen before. As is so common, a bug in one area of the system can have surprising impacts on security.

• Close file descriptors aggressively - before chrooting. We don't wish to leave handles open to non-jailed resources because these can all be exploited by those living inside the jail. Some file descriptors are required (day, to the syslog daemon), but developers should make a point to close anything that is not strictly required.

• Link config files from the outside - Some systems (such as BIND) share the configuration file between the jailed daemon and other utilities that are run from user mode. In this case, the config file simply must live inside the jail so that the daemon can access it, but the other utils from user mode still need to access this file. Rather than rebuild these utilities to use the special path (say, /chroot/named/etc/named.conf), instead go to the "regular" place for this file and create a symbolic link from the outside to the inside of the jail:

  # ln -s /chroot/named/etc/named.conf /etc/named.conf

  This allows most of the tools to operate "normally", though one has to be a little more careful that users editing /etc/named.conf realize that they're affecting a jailed system.

  This doesn't go the "other" direction, though it's not always obvious at first. Symbolic links from inside the jail to the outside will work for the administrator but will not work for the system running inside the jail.

EXAMPLES

Navigate: More Tech Tips

[Home] Stephen J. Friedl Software Consultant Tustin, California USA