Ask your kids. If your kids and your ex boyfriend (a normal human being) want to see each other than of course you should let them. You involved this person in their life for 9 years and they see him as a father figure. I'm sure you were happy that he was there for them and they actually cared for him so now I'm not sure how much of a right you have to take that father figure away from your kids. It's like snatching part of their life away from them and telling them it never happened but it did and it for because he was there.

Often times I see this becoming an issue when the mother wants to move on to another relationship but the ex boyfriend still being present in her kids' life makes the new person jealous. At that point you would need to assess your priorities and think what's more important to you. Is it more important that you have that person as your boyfriend or is it important that your kid's stay happy?

If your ex boyfriend isn't a normal human being and would hurt the kids than that's another story and I would keep them away from him.