Hmmm, interesting question, lots of nostalgia.
Aside from an old Trident PCI with 1MB of VRAM that was impossible to find a driver for Windows XP (confined to Standard VGA driver forever), I had a bad experience developing with a AMD 5770. I still consider the card very very good but had problems with the drivers to be more specific, it was a buggy D3D9 implementation that wasted almost 3 weeks of my life. It could have been any AMD card at the moment but it happened to be that one I had for development and that is why I must consider it my worst card.
Lots of mobile chips that had to hack drivers too.

I know that the subject is about worst GPU but I couldn't resist to honor the good fallen ones. I've have had many brands and models of video cards and I really can't say they were bad cards, they all served their purpose at the time and that is why I got them. From what I still remember:
- Diamond S3 Savage 4: UT99 with MeTaL renderer FTW! I absolutely loved this card.
- 3dfx Voodoo3 PCI: Unreal with Glide renderer FTW!
- 3dfx Voodoo5 5500 AGP: Used for developing Glide 3 but got it as a novelty after 3dfx had gone. I still have it.
- SIS 8MB ATI Rage based AGP: Used mainly for debugging PCs.
- ATI Rage 128 Pro: A really good card for the time. This one had TV output unlike many others.
- MSI Geforce2 MX: Played "Black n White" nicely.
- Geforce4 MX 4000: UT 2003...mmmmmmmmmmmmmmmister crow!
- After a few years with crappy laptop gpus I then jumped to the MSI Geforce GTS 450, Saphire 5770 vapor-x, eVGA Geforce 750 ti. I'm now settled for the low-mid range and will continue to do so in the future.