Until the mid-1800s, lobster was considered a terrible, low-class food. It was frequently served to prison inmates, and servants often had conditions in their employment contracts stating that they refused to eat lobster more than twice a week.

Some states even had laws against serving lobsters to prisoners more than a few times per week, since it was considered a cruel punishment, similar to being forced to eat rats.

Sources:
http://en.wikipedia.org/wiki/Lobster#History
http://www.gourmet.com/magazine/2000s/2004/08/consider_the_lobster?currentPage=2

For more, check out the story of How Lobster Got Fancy