i don't think proper rl fans are any less passionate it's just that rugby league is a relatively small niche sport. It hasn't been able to broaden its appeal significantly to the broader cross section of society in terms of class and multi cultures. A big part of this imo is the lack of focus on international competition. Supporting your country brings in all walks of life and sadly RL has never really been good at making the kangaroos, kiwis etc massive brands that non rl fans would get behind.
In terms of Sydney you have clubs that have grown out of small suburban catchments. Aside from London metropolis and Melbourne I struggle to think of any other city in the world with so many teams in one city. There are only so many sports fans and only so many of them will be interested in rl.
Finally throw in crap scheduling, crap weather, crap,stadiums etc and you get your current crowd avg which hasn't changed hardly in a decade of 15k or so.
The notion that Sydney is some megatropolis of culture, great winter weather and interesting things to do on a weekend so that is why people don't go to nrl is a furphy that doesn't back up when you look at cities like Milan, Barcelona etc who are much more of those things yet pack out their sports stadiums.