My mandarin wont fruit

Kate starts with ...
I have a more than 4 year old Imperial mandarin. It has grown well and is 2.5m tall however it still hasn’t fruited at all. Every year it flowers and we get tiny little ‘fruit’ on the end of the flower however the fruit then turn brown and fall off. I have 3 other citrus (lemon / orange and grapefruit) that all tried to produce fruit almost as soon as I planted them. We have a bee hive so pollination shouldn’t be an issue. Could it be something fundamentally wrong with the tree (is it possible to get a male mandarin) or do you think it could be a nutrient deficiency of some sort? Any advice would be much appreciated. Im getting to the point where Im thinking of cutting it down and trying something else.

Pictures - Click to enlarge
Picture: 1	Picture: 2	Picture: 3