bystander :
SSAA/FSAA, but you won't likely be able to handle that sort of AA levels, except on old non demanding games. It probably isn't even an option.
Of the ones you listed, TXAA is likely the best, but SMAA is something new, and may be similar. You'll have to test.
SSAA: Super Sampled AA
Your GPU renders the image at a higher resolution and down-samples the image with a filter, when the processing is done. Simply imagine a very sharp pointed edge and see it from a distance. It will look smothened [AA'd].
MSAA: Multi Sampled AA
The graphics card renders to a surface that is larger than the final image, but for shading a cluster of pixels, [that will later be cascaded into a single pixel] the pixel shader is run only once. We save a lots of fill rate, but we still use memory. This technique will not work on the effects coming from the shader [1x AA].
Also it will not work for a deffered renderer. Lighting effects are applied on each image post MSAA treatment.
CSAA: Coverage Sampled AA
MSAA optimised by N-Vidia. ATI has something similar too.
These are three basic AA filters.
Beyond 8X AA will not really show any noticeable difference, whatever might be the AA mode [assuming you are not playing GTA 4].
FXAA, like bystander said, is smoothing from the outside.
SMAA at 1X level is said to give GFX similar to 8X of standard MSAA.
Performance wise, if game AA is not good, SMAA is the best. Else 8/16X MSAA will do the job.
FXAA will be best for games like GTA 4, which do not have AA by default.
But all these are heavily dependent on the game you are playing. For Crysis, for example, SSAA will kill your PC, as it will AA every leaf of the nearby tree.
On the otherhand, in Battlefield 3, it might be alright, considering the boxy/ simple geometrical objects the GPU has to deal with.
Conclusion: Try out every setting, and try to find the best for your game. MSAA will not, in most cases, hit the performance till 4X. IF game engine doesn't perform AA well, inject SMAA and still performance will not be hit. Inject FXAA only if your PC is awesome and the game is not too demanding. Use SSAA beyond 8X, only after keeping your PC into a refrigerator.
Rick