There are many great answers here, and I'd like to add one thing that I don't think has been addressed - there's always more to do. ALWAYS.

I teach at the community college level, so I have it "easy" in some ways (almost never have to deal with parents, curriculum is less mandated from above, depending on my schedule I may only teach 2 hours in a given day though I might also teach 6 on others).

But the work is quite literally unending.
Have I finished grading this week's assignments and quizzes? Do I have the quizzes written for next week? Are my slides ready? Which activities are we doing? Do I have all the pieces I need for them? Did I make the assignments in the online learning system and write the rubrics? There's an exam in two weeks, have I finished writing the practice questions and the key? Have I finished writing the exam? What about that question a student asked in class and I said I'd get back to them because I didn't know the answer, have I researched that yet?

Once all the "actual job" stuff is done, there are the meetings - with students, department, division, all-faculty, with committees or single faculty members to work on a specific project. My courses are constantly being revised to be congruent with current information and closer to accepted best pedagogical practices. I'm also working on making my course material ADA compliant (because I never know when a hearing-impaired or sight-impaired student will sign up for my course and the compliance office has a sometimes-absurd amount of work to do already so may not be able to help me when I need to fix things at the last minute).

Did I mention pedagogical best practices? If I want to use the most effective teaching methods that I can, I need to keep up with the pedagogy literature (and I should also probably be keeping up with the scientific literature in my field, oops).

And I didn't mention conferences, professional development, and continuing education. Or outreach! Any "free" hour could potentially be filled with any number of things to help me become a better teacher or help the students have a better experience.

In a strictly teaching position at the post-secondary level, it is at least possible to actually "only" put in 40 hours a week and be done, though some of those hours will inevitably end up being at weird hours and many people will only be doing an OK job with 40 hours of work. At the k-12 level, especially once students start having homework to do, it certainly seems like everybody is spending well north of 40 hours a week. Yes, most of my mom's colleagues when she was teaching 5th grade were "only" spending 40 hours a week actually at school, but everybody brought work home, often 2+ hours of work a night on weeknights and then a big chunk of the weekend spent working, too. That means 6-7 hours a day of student contact (always on, always professional, always with an eye out for behavior problems) with 3+ hours of solo work, frequently with few or no breaks beyond the necessaries.

I want the best for my students, and I'm a fairly new teacher who hasn't taught these courses many times already; that means that I am continually in a state of wanting more hours to spend than there are waking hours in my day. Something has to give, so I cut it back to what feels like the minimum, and every quarter I hit a point around halfway to three-quarters of the way through the term where I realized I overdid it and something has to give. So I do my best and it isn't all that great anymore, where I take a week (or more) to grade assignments and quizzes, where my slides in class are walls of text instead of the pared-down slides with pictures I like, where there are a lot less activities and a lot more lecture. It's a crappy thing to do to my students, but it's better than me having a total breakdown the week before finals, so each quarter I do it and promise myself I'll try to do better next time.

For all teachers, especially for k-12 teachers, it seems like there is enormous pressure to be better and do better, from the administration, from the parents, from society at large, and from oneself. And there is literally always, always more you can do. So to take time to take care of yourself and *gasp* relax/recharge is hard. If you're not careful, it can start to feel wrong and evil to take time off. And that way lies burnout.