Even if it's only by a little, it proves that you're not reliable. People are counting on you as a team member.

The chances are that even if it all tends to work out fine, you not being punctual probably causes more hassle for your colleagues.