Using scientific methodology in marketing

Traditionally there is a perception about advertising and marketing being a field dedicated to arts, creative people thinking about slogans and filming videos for the next TV campaign.

Although this in some way remains true, there are some fundamental changes happening in the industry that influence the basis of any marketing activity.

An increasingly popular approach to make decisions in an organisation is using scientific methodology. What does this mean?

The scientific method is the way science has been accumulating knowledge over time based on evidence. Formulating hypothesis based on observations that can be proven by running a test. In a nutshell, this is a framework that will help us make decisions to our strategies and tactics to improve business performance with confidence.

This methodology has been adopted by online business, due to the nature of the channel where these operate and access to data, this has facilitated the adoption of this approach. Some very well known examples of organisations with a very strong culture around testing are Booking, Expedia or Google. Today this aproach is considered the standard in the industry. The benefit of using this approach is that it removes uncertainty and decisions based on gut-feel or Pay-grade and incentivising a data-driven culture based on results.

What can be tested?

You should be able to test anything. Some organisations are very scrupulous about this and will have very robust frameworks to make sure testing is implemented everywhere and run multiple tests on a daily basis. From a marketing and media perspective some examples are:

• Channels effectiveness
• Creative messaging
• Creative structure or format
• Creative colours
• Content in landing pages
• Content in forms
• Ad formats
• etc

What are the steps to follow?

The steps you have to follow to implement this methodology are: 1. planning your test, 2. running your test and 3 evaluating it.

1. Planning your test

Focus on identifying what changes or improvements with have a higher impact on your business, balance the expected outcome with the effort required. Use data and insights to define the tests you want to run and follow the next steps:

• Identifying an insight or observation: This could be from market research or insights we gather from our customers.
  • Ex: As a marketer, my goal is to identify ways to bring more revenue for the business. I have observed that the homepage of my website receives most of the traffic. I have observed, there is no clear call to action or button.
• Defining an hypothesis: We formulate hypothesis based on this observation as a variable. This usually has the format similar to: If we do this then that.
  • Ex: If I incorporate a call to action button to the homepage, I will increase the number of revenue by generating more leads.
• Defining a success metric: This will be tied to the the objective. We have to assure that the variable that we introduce will be influencing the outcome. (Conversions, engagement, increased volume of customers, conversion rate, etc).
  • Ex: Number of leads generated.
  • You can also use a secondary metric: Engagement (CTR, Time on site).
• Defining the testing methodologies: Here is where things get a bit technical. Essentially there are different methodologies from the most simple and A/B test to incrementality, pre post testing or multi-arm-bandit testing. Depending on the circumstances you’ll want to use a different methodology. Examples of circumstances you’ll be using each one of them:
  • A/B test – Define a and test. Control: the original homepage design. Test: the new page with the call to action button.
  • Pre/post – Not as reliable as an A/B test as results will be collected in different periods.
  • Incremental – Measuring the impact of activating a channel, used when there are too many variables in place to simplify it with an A/B test. For example: measuring the impact of activating a channel or program (eg:display advertising)
  • Multi arm bandit/Multivariant – Testing different elements on a creative and it’s combinations at the same time (Banner, landing page or creative).

Once you´ve followed the process you´re ready to take this into action.

How do you run your test? You can use many testing tools to run it, this depends on the type of test you’re looking to run;

Choose the platform you will use to run the test

• Media partner tools: Most media partners will have their own A/B testing tools integrated. Google, Facebook, Linkedin, and most email marketing platforms have.
• Website tests: There are a plenty of tools available and very easy to set up: Optimizely, Crazy egg, etc.

2. Run your test

Make sure the test is properly implemented and that you have set up a control and a test. Control being the sample of users that won’t be exposed to the changes you are doing or the ads that you are running. This way you have a way to measure the influence of your test.

3. Evaluating the results of your test

Understanding your results

Have in mind that to assure the results of the test you are running are relevant, you need to assure these are statistically significant and understand the confidence level of your test. There is a formula used to identify this point:

You can read in more detail a comprehensive explanation of each one of the elements of this formula. In essence, the output of this formula will provide you a result to determine if the result of your test is significant, or above the confidence level. This confidence level usually set at 90 to 95%.

This means that you will require certain amount of data to identify if your test is statistically relevant.

As a shortcut use a calculator to measure your confidence level, there are many resources online you can use. These calculator can also help you estimate how much data you will require to set up the test and in this direction estimate how much budget you should allocate to run it.

Framework to increase the use of testing in you your business

You should now have a better understanding of how to run a test. However, it won’t be very useful if the organisation you are working in doesn’t have a scientific testing culture.

How are you going to make this part of the business culture and implement the methodology in your organisation?

If you are a business running online, it is quire likely that some teams in the organisation are already using it. How can you improve the impact and the culture across your organisation.

• Make sure you promote this as part of the culture of your business through your values.
• Organise training sessions with specialists that can help you adopt this culture
• Organise a workshop where you capture ideas across the organisation about tests that can have an impact to the business
• Structure a backlog of tests that can run through the year (organise this backlog in a quarterly basis).
• Prioritise the backlog based on business impact / relative effort
• Reward the teams that do an effort to implement this culture by measuring how many tests and the impact of it.
• Allocate resources to hire data analysts that can coach and support the teams implementing a scientific methodology.

Once you understand the logic you’ll find this very easy to implement. Bear in mind most if not all online advertising platforms will come with some sort of testing tools or capabilities, making this much easier to implement

Are you already implementing this methodology in your organisation? What are the results that you have achieved in your tests. Please share these in the comments below.