=========================== R E L E A S E N O T E S =========================== Broadcom Advanced Server Program (BASP) Driver for Linux 2.2.x and 2.4.x kernel Version 2.0.2 June 28, 2001 Table of Contents ----------------- Overview Limitation Installation Install - TAR archive Install - RPM package Files Configuration baspcfg Startup scripts Configuration scripts Broadcom NICE patches Uninstall - RPM package Removal of physical interface in Generic Trunking mode Known problems Logs Overview -------- BASP is a kernel module designed for Linux 2.2.x and 2.4.x kernels that provides load-balancing, fault-tolerance, and VLAN features. These features are provided by creating teams that consist of multiple NIC interfaces. A team can consist of 1 to 8 NIC interfaces and each interface can be designated primary or standby. All primary NIC interfaces in a team will participate in Load-balancing operations by sending and receiving a portion of the total traffic. Standby interfaces will take over in the event that all primary interfaces have lost their links. VLANs can be added to a team to allow multiple VLANs with different VLAN IDs. A virtual device is created for each VLAN added. BASP supports Smart Load-balance, SLB and Generic trunking. In SLB mode, all the NIC drivers must support Broadcom NIC Extension (NICE). In this release, several NIC drivers patched with NICE are included. SLB mode works with all Ethernet switches without configuring the switch ports to any special trunking mode. Only IP traffic will be load-balanced in both inbound and outbound directions. The Generic trunking mode does not require NICE and can work with any NIC, however, it requires the Ethernet switch to support the technology and be properly configured. This mode is protocol-independent and all traffic should be load-balanced and fault-tolerant. Limitation ---------- BASP has been tested on Red Hat 6.2, 7.0 and 7.1. Following installation procedures work with these two distributions. Minor modification to installation scripts may be required if BASP is installed on other i386 Linux distributions. BASP also supports SuSe and Red Hat Linux for IA-64 architecture. Installation for SuSe Linux must be through TAR archive. Please follow instructions in "Install - TAR archive" section. Installation for Red Hat Linux can be through RPM or TAR archives. Refer to appropriate section for installation details. VLANs are only supported by the 3Com Gigabit Server NIC. As opposed to VLANs support in other platforms, e.g. Windows and Netware, VLANs are not supported by Alteon Acenic driver (acenic.c). Installation ------------ For users of Redhat 6.2, 7.0 and 7.1, please follow instructions in "Install - RPM package" section. For users of other Linux distribution, please follow instructions in "Install - TAR archive" section. For users of Linux for IA-64, please follow instructions in "Install - TAR archive" section. Install - TAR archive --------------------- BASP for Linux is shipped in mixed forms, where the platform and kernel specific files are in source code, and the core file is in object form. Three packages are shipped in this release: two tar archives and one RPM package. "basplnx-{version}.i386.tgz" is the tar archive for i386 platform, and "basplnx-{version}.ia64.tgz" is the tar archive for IA-64 platform. To uncompress and expand the tar archive, run % tar xvfz basplnx-{version}.{arch}.tgz The installation process involves the following steps: (1) Configure the major device number for BASP configuration. The default major number is 254 and it can be changed by modifying "Makefile". (2) To build kernel module, "basp.o", % make Note that the Make process will automatically build the correct module for different kernel flavors, e.g. symbol versioning and SMP support. There is NO need to define -DMODVERSIONS in the Makefile. (3) To create device file and to copy files, % make install (4) To update the module reference, % depmod -a (5) To load the driver, % insmod basp (6) Follow the Configuration section to set up the teams. Install - RPM package --------------------- (1) To install the RPM source package, run % rpm -i basplnx-{version}.src.{arch}.rpm (2) CD to the RPM path and build the binary driver for the kernel % cd /usr/src/{redhat, OpenLinux, turbo, ...} % rpm -bb SPECS/basplnx.spec Note that the RPM path is different for different Linux distributions. (3) Install the newly built package % rpm -i RPMS/i386/basplnx-{version}.{arch}.rpm The driver and other required files will be installed. (4) To load the driver, % insmod basp (5) Follow the Configuration section to set up the teams. Files ----- Makefile makefile baspcfg precompiled configuration utility bcmtype.h commonly use type header file blf.c BASP module entry points blf.h ioctl interface blfcore.h core interface blfcore.o pre-compiled core object blfopt.h automatic generated header file by Make blfver.h version header file nicext.h NICE header file pal.c platform abstraction implementation pal.h header for paltform abstraction release.txt this file nice-2.2.16 contains NICE enabled driver source code scripts contains sample scripts scripts/basp init script, goes to /etc/rc.d/init.d scripts/baspteam start/stop script, goes to /etc/basp scripts/baspif start/stop network i/f, goes to /etc/basp scripts/team-sample sample script of SLB team with 3 NICs scripts/team-gec sample script of GEC team with 3 NICs basp.4 man page baspcfg.8 man page for baspcfg utility Configuration ------------- The BASP distribution includes an utility program and several scripts for team configuration. The configuration process involves the following steps: (1) Copy a configuration script from "/etc/basp/samples" directory to "/etc/basp" directory. Note the configuration script must be prefixed with "team-". (2) Modify the configuration script to (a) change the team type (b) add/delete physical network interfaces (c) add/delete virtual network interfaces The syntax of the configuration script can be found below. (3) Manually start the team for the first time, % /etc/rc.d/init.d/basp start (4) For Red Hat Linux, run "netcfg" to configure the IP addresses of the virtual network interfaces. Note-1) Starting the team is only required for the first time, and the teams will be started automatically in subsequent reboots. Note-2) Forming multiple teams is possible by copying the sample files into "/etc/basp/team-" and modifying this file as described in the sample file. Note-3) In order to create more that one virtual interface (VLAN) for each team, refer to the respective description section in the sample files. baspcfg ------- "baspcfg" is a command line tool to configure the BASP teams, add/remove NICs and add/remove virtual devices. Following is the usage of this tool: baspcfg v1.2.0 - Broadcom Advanced Server Program Configuration Utility Copyright (c) 2000-2001 Broadcom Corporation. All rights reserved. usage: baspcfg commands: addteam -- create a team delteam -- delete a team addva [macaddr] -- add a virtual adapter to a team delva -- del a virtual adapter from a team bind -- bind a physical adapter to a team unbind -- unbind a physical adapter from a team show [tid] -- display team configurations where tid -- An unique ID for each team, starting from 0 type -- Team type: 0=SLB, 1=FEC/GEC tname -- ASCII string of the team vlan_id -- VLAN ID: from 1 to 4094, 0=untagged or no VLAN vname -- ASCII string of the virtual device macaddr -- MAC address (optional), e.g. 00:10:18:00:11:44 role -- Role of the physical device: 0=primary, 1=hot-standby device -- ASCII string of the physical device, e.g. eth0 Startup scripts --------------- (1) basp This script is intended to be installed in /etc/rc.d/init.d directory. After copying the script, run "chkconfig --add basp". This script will be executed at runlevel 2, 3, 4 and 5. When "basp" is run, it will search the /etc/basp directory to list all the files with "team-" prefix, and then it will invoke "baspteam" script to add or delete the teams. It is normal that each "team-*" file in /etc/basp represents 1 team. (2) baspteam This script is called by "basp" script to add or delete a team. To install, create "/etc/basp" directory and copy this script over. To manually add a team: % baspteam team-sample add To delete a team: % baspteam team-sample del Note that "team-sample" is the configuration script. (3) team-sample This script contains a SLB team configuration with 3 NICs: eth0, eth1 and eth2. The team name is "TeamSample". All 3 NICs are primary. One virtual interface is also created for this team and the name of the virtual interface is "sw0". "sw0" is the device that "ifconfig" should be run against to set up the IP address. VLANs are not enabled in this script. This script and "team-gec" are intended to be customized. Refer to the configuration scripts section for details. This script should be copied to /etc/basp directory and retain the "team-" prefix. (4) team-gec This configuration script creates a GEC team with 3 network interfaces, eth0, eth1 and eth2. The team name is "TeamGEC". All 3 NICs are primary. One virtual interface is added to the team with the name "sw0" and VLANs are not enabled. This script and "team-sample" are intended to be customized. Refer to the configuration scripts section for details. This script should be copied to /etc/basp directory and retain the "team-" prefix. Configuration scripts --------------------- Both "team-sample" and "team-gec" are configuration scripts that follow the same syntax, as following TEAM_ID: this number uniquely identifies a team. TEAM_TYPE: 0 = SLB, 1 = Generic Trunking/GEC/FEC TEAM_NAME: ascii name of the team TEAM_PAx_NAME: ascii name of the physical interface x, where x can be 0 to 7. TEAM_PAx_ROLE: role of the physical interface x 0 = Primary, 1 = Hot-standby. This field must be 0 for Generic Trunking/GEC/FEC team. TEAM_VAx_NAME: ascii name of the virtual interface x, where x can be 0 to 63 TEAM_VAx_VLAN: 802.1Q VLAN ID of the virtual interface x. For untagged virtual interface, i.e. without VLAN enable, set it to 0. The valid VLAN ID can be 0 to 4094. Broadcom NICE patches --------------------- Also included in this release are network device drivers patched with Broadcom NICE support. These drivers are originally taken from the Linux 2.2.16 kernel distribution. To installed patched drivers: (1) Copy the Broadcom NICE header file, "nicext.h", to the Linux kernel include directory, % cp /usr/src/nice-2.2.16/nicext.h /usr/src/linux/include/linux (2) Rename the original network device driver under the Linux kernel source tree, "/usr/src/linux/drivers/net". (3) Copy the patched drivers to the Linux kernel network driver source directory, i.e. "/usr/src/linux/drivers/net". (4) Follow the kernel rebuild instructions to configure kernel support for these drivers, i.e. % cd /usr/src/linux % make config (5) If the patched drivers are configured into the kernel, goto step (7). If the patched drivers are configured as modules, goto step (6). (6) In the case of supporting only the module version of these drivers, it is possible to simply run the following to compile patched drivers and to install them into the proper module directory: % make modules % make modules_install There is no need to compile the complete kernel. Goto step (8). (7) Rebuild the kernel to compile these patched drivers % make clean % make dep % make (8) Either reboot the system or unload/load the patched modules. Run configuration scripts to test the patch. Uninstall - RPM package ----------------------- To uninstall RPM package, % rpm -e basplnx-{version}.{arch}.rpm and to reboot the system, % reboot Removal of physical interface in Generic Trunking mode ------------------------------------------------------ In Generic Trunking mode, all the physical and virtual interfaces belonging to a team have the same MAC address. This MAC address is the same address as that of the first physical interface bounded to the team. In the case that this first physical interface is removed dynamically from the team using "baspcfg" tool and bounded to the protocol directly, this could lead to a duplicate MAC address problem on the network. Note that if the removed physical interface does not participate in any traffic, there will not be any problem. To properly remove a physical interface in a Generic Trunking team, follow the steps listed below: (1) Backup the original team configuration script % cp /etc/basp/team-gec /etc/basp/backup-gec Note-1) "team-gec" is the name of the configuration script. Note-2) "backup-gec" is the name of the backup script. The name of the backup script must NOT be prefixed with "team-". (2) Modify the team configuration script to remove the physical interface (3) Stop the running team % /etc/basp/baspif /etc/basp/backup-gec stop % /etc/basp/baspteam /etc/basp/backup-gec del (4) Re-start the team % /etc/basp/baspteam /etc/basp/team-gec add % /etc/basp/baspif /etc/basp/team-gec start Known Problems -------------- (1) We have identified that the network configuration utility, Yast2, distributed with Suse Linux 7.0 and 7.1, failed to save the IP address assigned to the virtual network interface. This problem also occurs when running under Caldera OpenLinux EDesktop 2.4 Kernel 2.2.14. The following error message will be seen when running the "basp" init script: Adding basp team-example OK Starting basp team-example FAILED To activate the virtual network interface, enter following command: % ifconfig sw0 172.16.10.10 up Consult ifconfig(8) for details in manually configuring a network interface. (2) DHCP address problem We have identified that there is incompatiblity between SLB and IP address obtained via DHCP. In the lab, we have seen that the virtual network interface lost the IP address after running for a period of time. To solve the problem, always assign static IP address to the virtual network interfaces. Logs ---- v2.0.2 - Added support to the Linux 2.4.2 kernel for IA-64 architecture. v2.0.1 - Fixed the server hang problem caused by removing a physical interface without first detaching from the team. - Added proc fs support for reporting team configuration. v1.2.2 - Enhanced "basp" init script to work with other Linux distrbutions. - Fixed jumbo frame problem. v1.2.1 - Added support to Redhat 7.1 release. - Added man pages. v1.2.0 - Added support to the Linux 2.4.0 kernel. v1.1.13 - Added check to prevent the removal of the primary physical interface in a generic trunking team. Such removal might cause duplicated MAC addresses on the network. v1.1.12 - Updated the release note. v1.1.11 - Fixed the problem of adding the same virtual interface twice. - Correctly attribute packet counters to the correct virtual interfaces when VLAN is enabled. v1.1.10 - Fixed the problem that invalid VLAN can be added but not removed. - Add removing of BASP teams in RPM de-installation. - Added usage and argument checking to basp script. - Fixed the problem that init script fails with multiple teams. - Fixed the problem in switching runlevels. v1.1.9 - Enhanced RPM packaging support. - Changed "basp" script to run at priority 11, which is right after "network" script. - Automatically detect the presence of "kgcc" and use "kgcc" to get around Redhat 7.0 compiler problem. - Added "baspif" script to start and stop network interfaces. v1.1.8 - Added RPM packaging. v1.1.7 - Fixed failure of server originated ping with VLAN enable. - Fixed loss of network connection under heavy traffic on SMP system. v1.1.6 - Changed both module and device name to "basp.o" and "/dev/basp" respectively for consistency. - Added startup scripts. - Enhanced Generic Trunking support by automatically programing the MAC addresses. - Addressed VLAN deletion issue. v1.1.5 - Fixed Generic Trunking problem. - Improved error messages in runing baspcfg utility. - Added Generic Trunking sample script. v1.1.4 - Added "show" command in baspcfg to display team configurations and status. - Improved module build process for MODVERSIONS and SMP kernel. - Display module build options in syslog during module load time. - Display blfcore.o version during module loading time. - Major changes in Makefiles to improve the build process. - Improved build documentation. - Changed to new baspcfg usage to the release.txt. - Added documentation to describe how to install NICE patched drivers. v1.1.3 - Fixed pal_read_mii() function. v1.1.2 - Fixed linux/if.h dependency problem which prevented BASP from detecting NICE supported driver on older kernels.