Lubuntu will be less than Xubuntu. But we're only talking 100-200 mb. Lubuntu aims to give you a good experience with lighter software, office, video and music players and such. For example abiword rather than the more robust libreoffice. So if it's for a hard drive install it's rather moot the size of the install with a modern drive.
They both strive for a lightweight desktop and which is faster is open for debate but abiword is likely going to run faster on a slow system with little RAM than libreoffice.
If you want something really small for a flash drive then you should look at puppy linux. I use it to work on old broken windows systems.