Union is just so competitive worldwide.
Here are all of the nations that the Wallabies have ever played, and their records against us (sorted into groups). There are 20 nations in all, because including sides like the Barbarians is really double counting as they play for one of the other teams on the list.
http://en.wikipedia.org/wiki/Australia_national_rugby_union_team
So we have NZ, South Africa and France, plus the British teams that give us a good game.
The next 4 nations can't beat us 75% of the time.
The bottom 10 of the so called rugby nations, haven't beaten us once, and none of them have played us even 10 times.
Get the list versus the All Blacks, it would be even more of a carve up.