I will never mess up Desktop version with Server version and vise versa.
This 5 cent tip comes out from real experiences and lessons I paid with couple days time on struggling to make a use of Server version as terminal only box.
It is true that you can always add Desktop packages on Server version or vise versa. But the result are quite different from respective versions, it has many problems and troubles.
Most people noticed that the difference of packages Server and Desktop versions have, but no one pay attention that these two versions have quite different configurations. The problems of miss-use versions caused by different configuration are elided or ignored.
The Server version is meant for server. Here are at least two things that affect you when you use Server version as your daily terminal console box: It is optimized for the server, assuming always has a wired network, always are power on (it does not reboot often). So your box will have much longer boot time. If you use WiFi only machine (most new style laptops do), the Server version of Ubuntu boot-up procedure can hang on there for 3 to 5 minutes, because lot modules on server depend on network when load it, but at that stage, WiFi network is not available yet, indeed, the Server version default setting doesn’t come with whole WiFi package.
These are only two little things that could catch you and give you headache. Try searching online for Ubuntu hangs on boot…. The solution for that is cumbersome and not clean. oh, I did not mention the users you created on Server version on terminal console won’t shown-up on Desktop login GUI if later you add Desktop packages, nor the GUI Users manage utilities. …. just another issue in the list.
So I will say Server and Desktop version are different enough that you do not miss-use them, although you can add packages to get certain features from other version, you will encounter problems caused by different configuration.
Although by saying that, I would suggest to use Desktop version for most of case, adding Server packages to Desktop version won’t affect much of your daily usage experience. You can easily change setting to disable GUI and use it as Terminal console. Use the Server version for server solely.
What we really lack is Ubuntu Terminal version: A version that is light, but optimized for daily laptop terminal usages only.