There's a few reasons for this. I'm going to assume that you're talking about a button-down shirt, where you'd normally be able to buy a "normal" one for, say $30 USD; whereas a "designer" one will cost about $150 USD (perhaps even more; I've seen white dress shirts that cost upward of $600 USD).

1. Quality of manufacture (and conditions of manufacture)

Yes, indeed, the quality is a big one. Even though it may appear to be "just cotton", there are many different grades of cotton. Have you ever felt pima cotton in comparison to regular cotton? The difference is substantial. Of course, you have to have a bit of experience with luxury goods to really be able to appreciate the qualities of better quality shirts - but once you get accustomed to it, there is no going back.

Then there's the manufacturing details. If you've got a slim build like I do, then you can definitely appreciate how much of a difference a truly "fitted" shirt makes. Part of the process of making shirts is establishing the measurements. The research that goes into finding the perfect width around the ribcage, stomach, etc., is all a part of the expenses of manufacturing a truly luxury shirt. Then there's the little details; extra stitches here and there to keep everything together, tighter stitches that are harder to do on a sewing machine, and perhaps even the positions of the buttons so that you can unbutton your shirt just enough to look good. All of that is what makes the difference between a "good" shirt and a "great" shirt.

Then there's the manufacturing conditions. Luxury shirts are usually made in Europe (less expensive ones usually in Bulgaria or Turkey; the more expensive ones are usually made somewhere in northern Italy). Aside from the fact that the workers here are typically much more skilled in making a truly luxury product, the quality of the working conditions are much better in these parts of the world, than there are in China, Bangladesh, Sri Lanka, etc.

If you're not familiar with how horrible it is, I suggest you check out this video:

To a lot of people, putting their money in "the right place" matters. If that means buying European-made clothes instead of clothing made in Bangladesh, and paying ten times more for it, then that's what it means.

That's quality overall; better materials, better workmanship, better work conditions. The garment inherently costs more to make - so it costs more to buy. But guess what? It lasts longer, looks better, and feels better. Which makes your $150, $200, $300 on a dress shirt an investment; not just a purchase.

2. The business expenses

Have you ever thought of why people perceive such brands as Polo, Gucci, Boss, Dolce and Gabbanna, etc., to be so luxurious? It's not just the clothing.

A large part of the business here is building up the "experience". Double-page spreads in magazines. Ads on TV. Product placement and sponsorship in golf tournaments, tennis matches and sailing competitions. All of this builds up your perception of the brand - and additionally, it costs money.

Guess who pays for this? Yup. The customer does. All of the legwork and expenditures that go into making the world think "we are a luxury brand" is a cost built into the sticker price that you pay when you buy these items from a store. While you do pay for the quality, you also pay for the prestige. While that may cost you a few hundred dollars in the store when you buy a shirt, it could well come back when you impress people with your sharp-looking Gucci shirt, and they know it's Gucci.

3. Exclusivity

If something becomes too available, it's no longer a big deal. Remember back in 2008, when the iPhone was a very new thing? It was a huge deal to have an iPhone. I remember people would be amazed when they see me talking on it, going "oooh, you have an iPhone?". Now, having an iPhone is relatively common, amongst people - and in many cases, it is expected (especially in certain corporate environments). Why? Because there are over 700 million iPhones that have been made; that's more than one for every ten humans on the planet. With that many floating around, it's a huge deal to no one to see that "this guy has an iPhone".

Luxury brands will fight tooth and nail to prevent this from happening. The reason why their clothing will routinely sell for so much is because it's a status symbol. If everybody was wearing a Gucci shirt, would it be a big deal? Of course not!

How do you prevent this from happening? Make buying a Gucci shirt too difficult for the average person. The fact is, they probably could make a profit selling their shirts at $85 or so. But then too many people would buy them.

Now, I'm sure you're thinking that it's bad business, right? Not really. Selling a thousand shirts at $85 generates about same amount of revenue as selling 300 shirts at $285 ($85,000 and $85,500 respectively).

When selling 70% fewer shirts, however, there are fewer shirts on the market, making it more exclusive, and therefore worth the money, for those to whom it matters.

You're paying more than the shirt is effectively worth, yes, but you're getting in return is a level of prestige that not so many people can have. To quite a few people, that prestige is priceless.