Instrumenting Customer Acquisition – SXSW Recap

Session by Michael Discenza – Capital Factory
March 10, 2017

  • Framework for data collecting and learning
    • Track KPIs to show health of business (dashboards and reporting)
    • Test hypotheses to guide effective action (based on KPIs)
  • Different goals for different stages
    • Early stage: establish product-market fit
    • Growth: maximize speed and efficiency of growth
    • Mature: maintain and optimize for profit
  • Posing effecting questions
    • Tied to business goals
    • Prioritize outcomes
    • Actionable path
    • Return on investment?
  • User funnel: Acquisition -> Activation -> Retention -> Revenue -> Referral
    • Maximize each segment via experiments and iterative improvement
    • Possible goals
      • Decrease customer acquisition cost
      • Increase lifetime value of customers
      • Improve customer satisfaction
      • Determine best messaging for a lifestyle segment
  • Scientific method applied
    • Research question: i.e. For my ideal customer profile, which messaging gets them to click the ad/post most often?
      • Messaging A: focus on convenience of finding live music with
      • Messaging B: focus on social aspect- finding / attending live music together
    • Hypothesis: i.e. messaging B would perform better
    • Experiment:
      • Only change 1 factor at a time – all else is the same (same channel too!)
      • Randomly assign independent variable
    • Data collection
      • Document process, decisions, timelines, etc in lab notebook so that other could replicate the process
      • Focus on conclusions: actionable, business implications, succinct
      • Statistical significance – is the sample size large enough?
  • Tools
    • PIWIK – open source pixel server
    • MixPanel v Google Analytics – choose MixPanel if retrieving your data later is important. Where there’s lots of data, Google Analytics is very expensive
  •  Some things to Google:
    • contextual bandits
    • counterfactual evaluation
    • Microsoft: decision service
    • Bayesian networks
    • Propensity scope matching
    • Machine decision automation -> predictive performance


Some pictures of key slides during the presentation: