http://en.wikipedia.org/wiki/Spatial_anti-aliasing
Antialiasing (AA) is a technique to reduce aliasing, which is an effect that results in visual artifacts (usually jagged lines) when trying to render high-resolution objects at lower resolutions. If you've ever looked at a diagonal line in a video game with AA turned off, you'll probably see that instead of a straight line it looks like a staircase - that effect is aliasing.
Antialiasing tries to reduces those artifacts by sampling objects at a higher resolution than the display resolution then downscaling the image to the display resolution while applying a subtle "blurring" effect to soften the edges of borders and make jagged lines less visible. The "x2, x4, x8, etc" you see in AA options is how much supersampling is used. AA x2 will supersample objects at twice the display resolution, x4 at four times, etc.
There are many different types of AA techniques that you will see in games. The most basic are SSAA (super-sample AA) and MSAA (multi-sample AA), both of which effectively require the GPU to render every frame at a multiple of the display resolution. Because of this they are both very performance intensive. MSAA is better performance-wise because it contains optimizations that allow it to only supersample certain components of the image. At 1080p, 4x MSAA is usually used as a "gold standard" by which to judge other AA implementations, both in terms of visual quality and performance cost.
There are newer AA algorithms that don't involve supersampling, like FXAA (fast approximate AA). FXAA in particular is merely a post-processing algorithm designed specifically to smooth jagged lines by using edge detection. Thus it has very little performance cost compared to MSAA. However it results in a global blurring effect (something you don't see with MSAA) and it may not reduce aliasing as effectively as higher-multiple implementations of MSAA (greater than x4). MLAA (morphological AA) is another post-process AA method that is has a low performance cost but also introduces blurring. Because these methods do not use supersampling, you will not see them listed with multiples (i.e. you will never see as an option "2x FXAA" or "4x FXAA"; it is always just "FXAA").
SMAA (enhanced sub-pixel morphological AA) is a hybrid of post-processing and supersampling AA methods that comes at an intermediate performance cost but without the blurring effect of pure post-process methods like FXAA and MLAA). And finally, TXAA is a new technique developed by Nvidia that adds a temporal element to the AA effect, reducing the "shimmering" effect you sometimes see with aliased images as you move the camera around. It is fairly performance intensive (Nvidia claims that 4xTXAA is faster than 4xMSAA but that has not been my experience) but does result in some nice-looking images.
When choosing an AA option, choose the one that gives you the best quality at a performance cost that is acceptable to you. With a GTX 780 you will probably be able to afford 4x MSAA at 1080p in most games. My suggestion is to start off at 4x MSAA (most games offer MSAA). If that lowers your framerate too much then go down to either 2x MSAA or something like FXAA.