A/B testing is a type of methodology that allows you to compare two types of design or communication in order to identify which one performs better.
This is a common method used in optimization and has roots in direct mail. Digital media professionals use this in a variety of ways, such as to test website design both for brand pages and e-commerce, advertising, or any piece of communication on the web. It helps professionals decide which type of communication will garner them more favorable results.
It involves testing one variable at a time (univariate testing), which means that only one element should differentiate the two types of advertising from the other.
These two types are presented to the same number of people within the desired target audience. The one that performs better based on a goal (ie. highest clicks), should be adapted.
Amazon Developers Site explains the math behind A/B testing.