Patching wget for fetching FTP "dot" files

I use the fantastic GNU wget tool to fetch data from remote web and ftp servers all the time, but it seems that the FTP fetch won't fetch filenames beginning with dot unless they are named explicitly. This means that the --mirror option won't really mirror a web site over FTP because the .htaccess and other "hidden" files won't be picked up. This is very annoying and I find it hard to believe that I'm the first to run into this.

I have modified wget-1.8.2 (and -1.9.1) to allow -. or --dotfiles command-line options (and the dotfiles keyword in the .wgetrc) to enumerate all files when doing a recursive fetch. This is done by sending the LIST -a command to the remote FTP server instead of the default LIST command.

I am using this patched wget as I write this to fetch about 3 gigabytes of data via FTP for a customer, and once I've gotten a bit more experience with these changes I'll submit them to the wget maintainers. I've not done a "full-service" change to a GNU tool yet and wish to make sure I've not broken something.

Building

These instructions will download the latest source, apply the patches, and install the whole thing.

# cd /source
# wget ftp://ftp.gnu.org/pub/gnu/wget/wget-1.8.2.tar.gz
# wget http://www.unixwiz.net/techtips/wget-patch.txt
# gtar -xzvf wget-1.8.2.tar.gz
# cd wget-1.8.2
# patch -b -p0 < ../wget-patch.txt
# ./configure
# make
# make install

Modified files in the wget build are:

src/main.c
src/options.h
src/ftp-basic.c
src/init.c
doc/wget.texi

Download

The patch files are on this web server:

wget-1.8.2-dotfiles-patch.txt (for 1.8.2)
wget-1.9.1-dotfiles-patch.txt (for 1.9.1)

Navigate: More Tech Tips