Definitely not a small club at all, Penrith has one of the biggest support areas, you start seeing Penrith fans from about Blacktown, Hawkesbury and Liverpool and then obviously more as you get closer to Penrith. That's a million potential fans from those areas alone but then outside of Sydney you see Penrith being widely supported in places like Bathurst, Blue Mountains, the entirety of the Central West, Dubbo etc so it has strong city support and regional support like the Dragons and Knights. Panthers would have 1.3 million people or so that could (Obviously not all of them will) identify with the club and be catered to via community outreach etc.
Bulldogs in comparison have an area of 300,000 people and no regional areas to get fans from, to the west they compete with Penrith, to the north they compete with Parra, to the east they compete with the Tigers or Roosters/Souths and to the south they compete with the Sharks. Bulldogs have a shit ton of money, that's why people view them as a big club.