I think the 'give a shit' factor comes into it, as it also does with when the fixtures are. The Australian guys ultimately care more about Origin over rep footy, with the emotional investment that is in state v state. Also, Origin benefits from being mid-season, not at the end when guys are largely cooked, and want a break.
Test footy would go way better if it were in-season. I don't know how you'd do that, and the NRL aren't likely to be interested, anyway. But you're going to get a better Test footy product when it's during the season, there is a window created for it, etc.
I know I'm part of the problem, I didn't watch the Pacific Cup last year and I can't remember the last Test I watched (2017 v Tonga, I think). But in November (as the Pacific Cup final was) I am over footy, I'm into cricket. If they truly want to make Test footy the pinnacle - and I doubt it could ever happen - they'd need to bring it in while the season is in full swing.