I decided that the only way to truly answer this question was to run the A/B test myself. Since our site is built on the Bootstrap 2.x grid, it was easy enough to enable/disable responsive by either including or excluding the Bootstrap responsive CSS. Using the andrew/split A/B testing gem for rack-based sites (read: Ruby on Rails), the setup for this test was dead-simple. Since https://CultCosmetics.com is an ecommerce store, our metric was conversion to purchase (completed orders). We ran the test over the span of several months in late 2013.

Sadly, the A/B test was inconclusive, showing a slight improvement with responsive disabled, although it was not a statistically significant difference. This is perhaps telling in and of itself, since I would have hoped to have seen a statistically significant result in favor of responsive design. Since we already have the responsive code built, and I prefer responsive myself, we now have responsive design enabled for all visitors. However, this makes me question whether, in the future, it is worth it to invest the time & effort to build responsive ecommerce sites.

Please note that +5.0% means a 5% relative improvement in conversion rate, not a 5% absolute improvement (ie: from 1% to 1.05%, not 1% to 6%). Also, this test was not specific to mobile/tablet visitors; it applied to all visitors. However, the test should not have made a noticeable difference to visitors on computers, so their conversion rate should have been the same in either scenario, and thus should not have skewed the results. Additionally, Google Analytics from the period of the A/B test shows that ~63% of our visits were on mobile devices/tablets, the rest computer (no, that is not a typo), so there was a noticeable difference for the majority of our visitors.