Tai O is one of the most beautiful spot of Hong Kong. No big buildings, no cars, just small houses built on piles, boats and a really short market. Really feels like country side on ocean coast. The cost of living there does not seem to be high at all. Inhabitants are living in a pretty poor condition. But do they need more than a simple life?
At about 1h30 from the center of HK (40min subway to Tung Chung + 50min bus, a bit tiering), you can have a great and fresh seafood lunch for cheap. I would recommend the spicy prawns and the squid!