Some caffeine will go through to your breastmilk, but if you are drinking it in moderation, it should not have much of an effect on your baby.
With that said, what is considered too much? According to KellyMom:
"Watching your baby is the only way to get an individual answer to this question...The estimate we often hear is 'less than five 5-oz cups of coffee' or 500 mg/day. Various sources suggest an upper limit for caffeine intake ranging from 300-750 mg/day."
Keep in mind, caffeine isn't just in coffee! See caffeine amounts in foods like tea, chocolate and more here. Also, always speak with your healthcare provider about specific ingredients that are safe when breastfeeding and pumping.