Thanx Kewlx
I found some informative tidbits:
"There are a number of ways to use 2 Ethernet ports :
Two Network Connections (two subnets)
Will allow your PC to be connected to two networks at the same time.
Balance-rr (Round-Robin)
Transmit packets in sequential order from the first available slave through the last. This mode provides load balancing and fault tolerance.
Active Backup(Fail Over)
Only one slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The bond’s MAC address is externally visible on only one port (network adapter) to avoid confusing the switch. This mode provides fault tolerance.
Balance XOR
Transmit based on the selected transmit hash policy. The default policy is a simple [(source MAC address XOR’d with destination MAC address) modulo slave count]. Alternate transmit policies maybe selected via the xmit_hash_policy option. This mode provides load balancing and fault tolerance.
Broadcast
Transmits everything on all slave interfaces. This mode provides fault tolerance.
IEEE 802.3ad
Creates aggregation groups that share the same speed and duplex settings. Utilizes all slaves in the active aggregator according to the 802.3ad specification. This mode provides load balancing and fault tolerance but requires a switch that supports IEEE 802.3ad Dynamic Link Aggregation with correct LACP mode configured.
Balance-tlb (Adaptive Transmit Load Balancing)
Channel bonding that does not require any special switch support. The outgoing traffic is distributed according to the current load (computed relative to the speed) on each slave. Incoming traffic is received by the current slave. If the receiving slave fails, another slave takes over the MAC address of the failed receiving slave. This mode provides load balancing and fault tolerance.
Balance-alb (Adaptive Load Balancing)
Include balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special switch support. The receive load balancing is achieved by ARP negotiation. The receive load balancing is achieved by ARP Replies sent by the local system on their way out and overwrites the source hardware address with the unique hardware address of one of the slaves in the bond such that different peers use different hardware address for the server. This mode provides load balancing and fault tolerance."
-attributed to Iggster http://www.techpowerup.com/forums/threads/using-dual-gigabit-ethernet-ports.118632/
Here are some other good ones:
http://archive09.linux.com/feature/133849
http://www.overclockers.com/forums/showthread.php/596734-What-do-you-use-your-dual-ethernet-ports-for
It appears you can do much with multiple ports on a NIC, sweet.