Machine Learning: The Art of Explore vs. Exploit – SXSW Recap

Session by Oscar Celma (Pandora)
March 11, 2017

Pandora’s mission: be the effortless source of personalized music enjoyment and discovery

Some mind blowing statistics at Pandora

75M monthly average users
24 hours of listening per month
75B thumbs-up
12B stations created
98% of artists spinning every month

How does Pandora decide what to play next?

Content based algorithm: music genome data
Collective intelligence: mining user behavior
Personalized filtering: your thumbs up and skips
Ensemble recommender: piece together output from 75 different algorithms

Challenges: balance familiar with unfamiliar

Exploit: play awesome music now. Tomorrow? Who cares. Don’t play music I don’t like.
Explore: play something risky. Learning what to play. Don’t play too many WTF (“what the freakommendation” – Paul Lamere“).

Novelty versus relevance

Exploit: low novelty, high relevance
Explore: high novelty, high relevance
Popular: low novelty and low relevance
Risky: high novelty, low relevance

How does Pandora test new ideas?

  1. Dream idea
  2. Experiment in small group (1% of users)
  3. If successful, roll out 6-12 months later

Metrics: did it bring new listeners? Did it avoid churn? Did they listen for longer?

Retention: time spent listening, active days

Activity: thumbs, skips, create new stations

Pandora’s Tech Stack (some of it)

Memcache, Redis, Python, Java, Scala, Hive, Spark, PostgreSQL, Hadoop (HDFS)



Founder Counseling: 7 Steps to Founder Success – SXSW Recap

Session by David Mandell (PivotDesk), Jenny Fielding (TechStars), and Nicole Glaros (TechStars)
March 11, 2017

65% of startups fail due to people issues. It doesn’t have to be that way!

1. Cofounder issues

  • Know who you’re going into business with
  • Learn how to take ownership of problems – “it wasn’t what he was saying, it was how I was hearing it”
  • Equity split: any animosity will grow over time
    • Vesting: Get it right at the start. Protects founders, not punishs. Investors would set this up anyway.
    • Split correctly when everyone is feeling good
    • Venture Deals (book)
  • Cofounder dynamics affect culture for the rest of the company
  • Difficult Conversations (book)

2. Find great mentors

  • Mental / emotional support
  • Who you can talk to even when you can’t talk to anyone else
  • Someone you can be vulnerable with
  • Pro tip: don’t call them a mentor (at first)

3. Intellectual honesty

  • “Shit that scares me to death” portion of quarterly board update. Shows you’re trying to be real. What’s being done to address the issues? Maybe your investors have insights.
  • Honesty with self

4. Direct feedback

  • Not become defensive when receiving feedback
  • Usually given by people who care about you and your business

5. Right investors

  • Understand their motivation and goals
  • Can make your life a living hell if they want to
  • If you receive multiple term sheets, go after best fit, not best valuation
  • Accelerator? Do it as early as you can. Helps plug holes in your business. Talk to alumni from the accelerator to get a preview

6. Passion

  • Were your born for this?
  • Early on: enthusiasm & passion > skill
  • Scaling the business: skills matter more

7. Self improvement and insatiable curiosity

  • Always ask what you could be doing better

Fierce Conversations – SXSW Recap

Session by Susan Scott, Author of Fierce Conversations: Achieving Success at Work and in Life One Conversation at a Time
March 10, 2017

This was one of my favorite sessions at SXSW 2017. I will be buying Susan’s new edition of Fierce Conversations when it hits the bookstores May 2. Mark your calendars.

As we continue to see technology accelerate at an exponential pace, the human element remains largely the same. There is no Moore’s Law governing human-to-human interaction, yet many of us invest far too little time here.

As a software developer its easy to obsess over building a resume lined with cutting edge technologies or the latest tech fad. But what about building the relationships around us? How brilliant could we be if we don’t have the emotional capital to share it?

Susan proposes we do this one conversation at a time, where the relationship is the sum of conversations shared or the conversations missed. As Susan points out, “The conversation is the relationship. How we spend our days is how we spend our lives”.

How do we come out from behind ourselves and make ourselves real? Fierce conversations serve one or more of these functions:

  1. Interrogate reality
  2. Provoke learning
  3. Tackle and resolve tough challenges
  4. Enrich relationships

“The person who can describe reality without assigning blame will always emerge the leader”

Instrumenting Customer Acquisition – SXSW Recap

Session by Michael Discenza – Capital Factory
March 10, 2017

  • Framework for data collecting and learning
    • Track KPIs to show health of business (dashboards and reporting)
    • Test hypotheses to guide effective action (based on KPIs)
  • Different goals for different stages
    • Early stage: establish product-market fit
    • Growth: maximize speed and efficiency of growth
    • Mature: maintain and optimize for profit
  • Posing effecting questions
    • Tied to business goals
    • Prioritize outcomes
    • Actionable path
    • Return on investment?
  • User funnel: Acquisition -> Activation -> Retention -> Revenue -> Referral
    • Maximize each segment via experiments and iterative improvement
    • Possible goals
      • Decrease customer acquisition cost
      • Increase lifetime value of customers
      • Improve customer satisfaction
      • Determine best messaging for a lifestyle segment
  • Scientific method applied
    • Research question: i.e. For my ideal customer profile, which messaging gets them to click the ad/post most often?
      • Messaging A: focus on convenience of finding live music with
      • Messaging B: focus on social aspect- finding / attending live music together
    • Hypothesis: i.e. messaging B would perform better
    • Experiment:
      • Only change 1 factor at a time – all else is the same (same channel too!)
      • Randomly assign independent variable
    • Data collection
      • Document process, decisions, timelines, etc in lab notebook so that other could replicate the process
      • Focus on conclusions: actionable, business implications, succinct
      • Statistical significance – is the sample size large enough?
  • Tools
    • PIWIK – open source pixel server
    • MixPanel v Google Analytics – choose MixPanel if retrieving your data later is important. Where there’s lots of data, Google Analytics is very expensive
  •  Some things to Google:
    • contextual bandits
    • counterfactual evaluation
    • Microsoft: decision service
    • Bayesian networks
    • Propensity scope matching
    • Machine decision automation -> predictive performance


Some pictures of key slides during the presentation: