We also struggled with the question. Our first daughter, while an amazing little miracle to us, was also an incredibly difficult baby. I had her at 33, and around 35 we started to talk about a second child, and I wasn't sure if I was physically up for it again. The deciding factor, though, was when my 3yo started to beg for a little sister. While there are times that I put myself in timeout because they are CRAZY (see example below), I wouldn't change it for anything. It completely changed the dynamics of our family, and I think made us better parents. Plus, our second was an absolute dream baby who slept and ate right on schedule. :)

Bringing baby #2 home:

Four years later still best buds:

Example of crazy:
This morning after screaming bloody murder the entire time I washed and dried Lily's hair, she looked in the mirror and told me how beautiful her hair looked. When I mentioned the crying fit she had, without missing a beat she replied, "Well, mommy, the tears are what made my hair look so beautiful." Here's her great hair: