No, it's not racist so long as the following are all true:
1) you ask this question equally of people who speak with no accent from every race
2) you have a reason to ask this question besides curiosity and "to avoid assumptions"
3) you have a good follow-up to their answer that's not stereotypical

With regard to 1: people pose this question to people with accents because the accent shows that they're not from the area. So the thinking is: hey, you have a British accent; you were probably raised in Britain. Likewise, inherent in asking a nonwhite person with no accent this question is the thinking: hey, you are nonwhite, you were probably raised somewhere else. It's not racist in the "i hate you because you're not white" sense but you are treating that person like an outsider because they're from a different race. And I consider that racism. So if you're going to ask the question, ask if of EVERYBODY equally - that is to say, add white people without accents to your list of answerers. You must have a good reason to ask it of minorities so it should be equally important to you to ask it of white people, right?

2. "Curiosity" is not a good enough reason to ask a question. There are thousands of questions we would be curious about that we don't ask because they are considered rude, including "Are you pregnant?" "Are you deformed?" and "Are your breasts real?" Curiosity, even with good intention, is not good enough reason to do something.

Likewise, "not wanting to make assumptions" is also inadequate as a reason. Yes, it's offensive when people squint their eyes and say "konnichiwa" to me but it's equally as offensive when people squint their eyes and say "ni hao" to me. If you have a non-racist reason to ask besides curiosity and besides "not wanting to make assumptions" I would seriously be interested in knowing that reason.

3. I get asked "where are you from" ALL THE TIME. More often than I get asked my name. My answer: I'm Chinese. Their response: "You don't look Chinese."
Sometimes the conversation continues with follow-up like "Do you know [this Chinese person]?" (because there are SO FEW chinese people in the world") or "Ooh I've been to China before" (now you're assuming I've been to China, probably assuming I grew up in China, again making me feel like an outsider). I guess the real problem is that people who look like me get asked this question hundreds of times in their lifetimes and the conversations are ALWAYS awkward so this question always makes me want to leave the conversation ASAP. I don't think I'll convince anyone to stop asking this question, but I think you should know that if someone is nonwhite, this person probably gets asked this question a lot, that not all nonwhite people like being asked this question and that it's generally not a great conversation starter, unless that person is from a different country. If you're actually a foreigner to the U.S. then you might enjoy talking about your home country. If you're native to the U.S., you might find it weird being asked to talk about a country that someone believes is your home country when in actuality it's not.

I've tried telling people I'm black but the people who ask this question don't think that is funny.