They tried to explain this in the movie, changing the past doesnt change the future, it creates an alternate reality that splits off from the main timeline
Was that when Banner was speaking with the Monk chick? Is so didn't she say explicitly that doing so was a bad thing and why she wanted the stone back? And if so then why bother taking all the stones and the hammer back