These little chaps are ace. Cheap and cheerful minibuses which bomb around, almost at your convenience. They're usually 16 seater minibuses which operate on set routes; unless they're the ones with the red tops, which go where they go - these are recommended for locals only.
These minibuses stop at bus stops and will also stop if you flag them down or if you tell the driver that you want to stop and get off. If you don't know the Cantonese for stop or the name of the place you want to stop, then make sure sit there looking slightly confused and lost, in the hope that a kindly stranger who does speak Cantonese will ask you where you want to get off and tell the driver for you.
Phonetically speaking, "lee dough, um goy" means "stop, thanks", although being able to say the name of the place you want to stop and thanks is preferred.
As buses and trains stop operating at around 11pm, these buses can also prove invaluable for the late night traveller. If I'm in Central and I miss the last underground train across to Tsim Sha Tsui, so I can get the last overground train back up to the New Territories, I can either get a taxi back and pay a fortune, or I can find the minibus in Wan Chai, which I know will get me up to Tai Po and then I can get a taxi from there. A much cheaper solution.
Buses are a cheap and cheerful way of getting round Hong Kong but, unless you know your way around and know the services well, I'd exercise an air of caution. After all, you don't want to get lost.
Unless you're travelling a well worn route, I'd only use buses for planned trips. There is a Route Search on the Transport Department website, which will tell you the fare and the duration of the journey. What's also rather nice id that if you follow the links to get more details about the route, it will show you photographs of the bus stops, which can be handy.
MTR - Trains
There used to be MTR (underground) and KCR (overground) services, but they both fall under the banner of the MTR these days. Generally, I travel to and from the New Territories on the overground service, then I move underground when I hit Kowloon Tong. There are route maps at every station, but here's a handy PDF for your reference.
As mentioned before, when travelling at night, be sure to check when your last train is. Oh, and if you're travelling on the overground service, you'll notice that there's an extra Octopus card swipe machine and markings for the First Class carriage. If you're travelling a good distance and you want a seat, then First Class might be a sensible option outside of rush hour, when it is likely that you might have to stand. It will probably double your fare, but it's pretty cheap to travel around, so that shouldn't be an issue, unless you're on a penny-pinching budget. Don't try and ride First Class without swiping though, as you could well end up paying a HK$500 fine for the privilege.
As well as being able to catch ferries to the outlying islands of Hong Kong, no trip should be complete without making the crossing from Tsim Sha Tsui to Central on the Star Ferry: it's cheap, provides a great view of both shorelines and is a very convenient way of hopping back and fourth between the island and Kowloon.
In my own experience, taxis are a last resort when it comes to getting around Hong Kong. They're the most expensive mode of public transport and not always the quickest, depending on where you're going. I have also had taxis refuse to take me on short trips, where I've had very heavy items or luggage with me, even to the point of having them feign ignorance / misunderstanding and drive off after stopping. Although, I have also had many perfectly acceptable taxi rides as well.
One thing to note, if you are taking taxis, especially over some distance, is that urban based taxis are red, New Territories taxis are green and Lantau taxis are blue. So, if you are heading from one area to another, try and get in a taxi which is returning from whence it came. So to speak.
Any how, unless it's very late and I don't know where to get a minibus from or I'm carrying more luggage than I'm comfortable taking on buses, minibuses or trains, then I won't get a taxi.
Now, I'll admit i don't use them as much as I should and that I've never taken the Peak Tram, but trams are a super handy and very economical way of getting around the northern side of Hong Kong Island.
If you're wanting to explore via the tram, then the 'Ding Ding' Hong Kong Tram Guide is well worth looking at.