There are dvd player which support the playback of wmv, mp4 and divx avi. However, as you said that you have a lot of videos, using cd will be a headache because it is slow and also it will not be cost effective because of the large amount of cd you need to buy. I am not sure if you can get one in your location, but I used to use a cheap maya dvd-405 player (about $45 AUD) back then with external usb HDD (not included).
The 500GB external HDD I got back then was formated to fat32 (the dvd player cannot support NTFS), plug HDD to the dvd player via usb and plug the video out of the player to the TV. As long as an individual video is not larger then 4GB, there is no problem (didn't run into that problem since full sd movie is about 700MB).
The dvd player lasted about 2 years then die and I decided to convert an old computer to a HTPC, the HDD is reused in the build. Can't complaint cos the player was so cheap.