Assessing The Effectiveness Of Communication Methods and Content Through A/B Testing

Client Background

Client is a mobile app based FinTech organization in the Indian market, operating in the space of personal loans, housing loans, health insurance, and mutual funds. Its aim is to make personal finance simple and accessible to a wider set of audience.

Business Objective

Client is a start-up company in their growing phase. As part of the journey, the client was working on figuring out different things that work best in the market and kept experimenting to acquire this learning. We worked with the Growth and Marketing teams, and the business objective was; to identify how efficient are different modes of communication (SMS/ Push Notifications/ Emails/ Performance Marketing) and what type of messaging content helps in higher customer engagement.

Solution

1. We leveraged data from different customer engagement and communications platforms like MoEngage, AppsFlyer, Kaleyra and Gupshup; which were being used by client
2. The, we performed series for A/B tests to assess the performance of different modes of communications and different types of messaging contents
3. The user base was divided into different cohorts with an uniform distribution across the cohorts based on user attributes (demographic and bureau info)
4. Different success metrics were defined, to track the effectiveness of these campaigns, viz. user reachability, user engagement (views/clicks) and user conversions (from on-boarding to conversion journey)
5. User engagement via clicks was studied using AppsFlyer data, which helped in tracking the engagement across different deep links that were incorporated into these campaigns
6. Defined the attribution time window to link any success event to a particular event, by looking at the data distribution and studying industry standards
7. Built a dashboard on Superset (which was later migrated to Tableau), to help the different stakeholders keep a track of the performance of these different campaign experiments over a period time

Outcome

1. These activities helped the clients to identify the appropriate communication channels and right set of content that worked best for different set of customers. This helped clients in enhancing their marketing strategy and channelizing their budget accordingly.
2. Success metrics were defined in terms of proportion of users who engaged on the platform after receiving communication and then till what stage of the product journey did they proceed. And eventually how many of them ended up getting converted to paid customers.