Andrew Chen has an interesting blog post up the dangers of the over use of A/B testing in product development.
In the commercial projects I’m working on I’ve been very much a proponent of testing game features, marketing material, etc. in a scientific way. As developers close to a project it is often very hard to know if what you are working on is going to resonate with your target market or not. When making a decision on a game feature with a passionate group there are often multiple view points about what is best and without a clear way to evaluate such decisions often the strongest personality in the room gets their way. This is designing by instinct and discussion. That isn’t always going result in the best decision for the game, hence my preference to use math and analysis to decide if it can be applied.
Testing variations of a feature against a large enough audience should ideally begin to reveal data that can concretely improve a feature if the test is designed correctly. As Andrew points out though this can result in overcompensated risk aversion and is better suited for small scale decisions rather than setting high level product direction.
In terms of game features specifically, I’m starting to incorporate more metrics into my applications so this type of data can be tracked by a database and data mined at a later time. I think this kind of tracking and optimizing of individual game features is a strong way of coming to a decision on game features that have no clear “best” solution.