I'd say the issue is that if you build an economy which is reliant on perpetual growth, a population that doesn't at least reproduce itself becomes a major issue.The problem (if there is one) isn’t immigration to Australia, rather it’s how to get immigrants (or just Australians in general) to not move to the big cities and to try and de-centralise Australian population centres, because at the current rate, it just does not seem sustainable.
It's why I don't think that it's inconsistent for economic conservatives to encourage immigration, or for the the traditional left economists to want to limit numbers. Just another example where people can have conflict between their social and economic political leanings.