ok, for starters you are wrong, as the discussion of a god is a religious philosophy by it's own nature. the idea of a supreme being is a religious concept. if you believe in any sort of god, you are religious to a point. HOWEVER... i think what you are talking about is Deism - the belief that a creator existed, but without all the supernatural religious mumbo jumbo attached.
the problem i have with Deism is that, by definition, there is nothing you can experience, or observe that would ever confirm nor deny the existance of a creator (much like standard Theism, at least until the end times anyway). furthermore - with Deism it doesn't even matter if there is or is not a creator... he created and left us here. believing it exists is null and pointless, because it's not watching and doesn't give a shit about what we're doing.
another criticism i have of Deism is that it only exists to answer a single mystery - how it all began. belief in Deism is absolutely useless and non-productive. every Deist would be far better served trying to answer that question with observation and science, rather than an unprovable, untestable hypothesis that has absolutely no impact on anything.
so my irk with that point of view is purely that it's useless. you believe that there's a creator but not in any of the supernatural stuff that usually comes with it. what's the point? why not just believe that the universe started through a provable, scientific process - because there is far more observable evidence to suggest that than anything else.
of course, if you believe that there's any sort of heaven or anything, then you very quickly roll back into the completely fabricated religion zone. you can't have it both (or more) ways.