Built to keep Playing: Analytics Strategies for Gaming and Entertainment companies
Hosted by Loops, May 20, 2025
At this virtual executive meetup, analytics leaders gathered for an intimate, high-impact session led by Tom Laufer (CEO, Loops) and Ken Rudin (Executive Growth Advisor, ex product at Zynga, Google, Meta, and ThoughtSpot).
Key talking points included:
Correlation vs. Causation in Farmville Ken Rudin shared an experience from his time at Zynga with the game Farmville. The team initially focused on increasing the number of friends players had, based on the correlation that more friends led to more playtime. However, AB testing revealed that this correlation was not causal; the number of engaged friends, not the total number of friends, was the key driver of playtime. This highlighted the mistake of relying on correlation without establishing causality through experimentation.
Experiences with Player Interaction:
An attendee resonated with the Farmville story, noting that having a large friends list doesn't guarantee engagement; meaningful interactions are more important for retention. They are currently experimenting with ways to nudge positive interactions between players sooner, even if they are not yet "friends".
Another shared their experiments with social presence, where they found that interactions within clubs, even small ones that provide mutual benefits, significantly improve retention.
Ken Rudin emphasized the importance of nudging players to interact in context, at a moment when they are likely to be receptive to the suggestion.
Another attendee agreed and discussed their approach to identifying attributes that encourage players to stick together after a match, even after a loss, to foster long-term engagement.
Engagement vs. Acquisition An attendee from the music industry noted a similar focus on engagement and retention over acquisition, aiming to increase user interaction with content beyond just streaming, such as merchandise and concert tickets. Ken Rudin shared his experience with YouTube Music, where a key differentiator was the ability to quickly provide personalized music stations based on users' existing YouTube watch history, leading to a faster "aha" moment and improved stickiness.
The Shift Towards Causal Inference Ken Rudin discussed the evolution of analytics tools, highlighting the significant advancement of causal inference, which allows for virtually running AB tests on existing data to infer causal relationships quickly and with high accuracy (around 90%). He emphasized that this approach saves significant time compared to traditional correlation analysis followed by extensive AB testing.
Micro-segmentation and Next Best Action Ken Rudin advocated for moving beyond broad user personas to micro-segmentation, which focuses on segments of size one to optimize engagement. The goal is to determine the next best action for each individual user based on their complete history, similar to a chess player analyzing every move to decide their next one.
Proactive Data Culture Ken Rudin stressed the importance of data teams adopting a proactive role rather than just reactively answering questions. A proactive data team identifies opportunities and recommends actions based on data analysis, becoming a driving force for the company, with product managers and engineers also leveraging these tools.
Proactive Insight Delivery and Natural Language Querying Ken Rudin highlighted the benefits of proactively sending out insights to relevant teams instead of waiting for requests. He also noted the growing capability of asking data questions in natural language, enabling more individuals within a company, such as product managers and engineers, to perform analysis without needing complex coding skills. He anticipates a future of "vibe querying," where users can have ongoing, conversational interactions with data systems.
Questions and Challenges Faced by Attendees An attendee reiterated their challenge of building an LTV model due to fragmented data across platforms, requiring significant data cleansing and integration efforts. They are exploring solutions to get real-time answers to business questions using natural language, which would be beneficial for non-technical marketing users. Tom Laufer commented on the iterative nature of LTV modeling and the trend towards using more sophisticated, less explainable but more accurate data science models for marketing optimization.
The Chaotic Nature of the Gaming Ecosystem Tom Laufer described the gaming and entertainment ecosystem as highly chaotic, with numerous interconnected factors making it difficult to balance various KPIs like retention and monetization. He provided examples such as adjusting game difficulty and payout ratios, noting that optimizing for one KPI often affects others. Companies sometimes operate in silos, focusing on a single metric without fully understanding the trade-offs . Live ops, with constant game changes and promotions, further complicates this ecosystem.
Challenges for Analytics Teams and Data Science Models Tom Laufer pointed out that analytics teams spend significant time analyzing experiments and repeatedly addressing evolving segmentation needs. Additionally, data science models in this vertical often have a lower deployment-to-impact ratio compared to other industries, possibly due to not accounting for external factors and network effects.
The KPI Tree for Managing Trade-offs Tom Laufer presented the concept of a KPI tree as a solution to manage the complexity and trade-offs between different KPIs. This visual representation maps how KPIs affect each other through correlation and causation, providing a live view of the impact of changes and enabling sensitivity analysis for planning. One attendee volunteered they had tried to build such a system manually, focusing on DAU and ARPDAU, and also attempting to create KPI trees for individual features manually.
Impact Analysis and Estimation An attendee explained that the goal of their process is to clarify the conceptual origin of impact for new features, aiding in rough estimation and providing insights for pitching to executives . Tom Laufer and Ken Rudin acknowledged the manual methods some companies use, with Tom Laufer suggesting the need for more scalable solutions due to changing KPIs.
Self-Serve Analytics with Vibe Querying Tom Laufer introduced "vibe querying" as an approach to self-serve analytics, enabling business and analytics users to gain insights without relying solely on dashboards. This method, exemplified by Loops Scout, allows users to ask questions in natural language and receive AI-driven analysis, potentially automating many repetitive queries and facilitating deeper research.
Experiences with Natural Language Analytics Interfaces Ken Rudin inquired about experiences with natural language interfaces for analytics. An attendee shared that while they have used some tools, a significant challenge is data governance and access rules that can limit the context provided by AI assistants. Another attendee echoed the data governance concerns but expressed interest in the potential for bespoke analysis to shift teams from reactive to proactive approaches.
The Importance of Context in AI Analytics Tom Laufer emphasized that context is crucial for vibe querying solutions to provide accurate and relevant answers, drawing a parallel to the onboarding process for data analysts . He noted that while some blame the AI, the quality of its output reflects the underlying data, modeling, and literacy, and that providing sufficient context is key. An attendee added that these tools can be powerful for analysts and data scientists who have the historical context to build upon the AI's output. Tom Laufer agreed, highlighting how data governance can define the scope of questions business users can ask.
Automated Root Cause Analysis Tom Laufer discussed the increasing trend of companies automating root cause analysis in the gaming industry to address the chaotic nature of identifying factors affecting KPIs like revenue. This automation can streamline daily, weekly, and hourly business reviews, saving time and reducing the frequency of urgent investigations.
Opportunity Sizing with Causal Models Tom Laufer presented the use of causal models and simulation to perform sensitivity analysis, illustrating how improvements to input metrics can affect output metrics, which helps identify significant opportunities worth pursuing. He pointed out that many hypotheses fail because the potential impact is not substantial enough, and this approach offers a more robust method than traditional frameworks like ICE.
Measuring Impact Without A/B Testing Tom Laufer explained how causal models can be used to measure the effect of interventions, like live ops or promotions, when A/B testing is not feasible. By simulating a control group, they can estimate what would have happened without the intervention, allowing for the identification of the real causal effect with a high degree of accuracy.
Typical Analysis Using Causal Models and KPI Trees Tom Laufer outlined common analytical use cases leveraging causal models, including understanding drivers of long-term retention and monetization, measuring temporary effects through pre-post analysis, and analyzing the impact of game economy changes using KPI trees . He noted that focusing on these areas typically yields the most significant benefits for companies.
Identifying Activation Metrics for Retention An attendee raised the challenge of identifying the initial actions (activation metrics) that drive long-term retention, such as D7 or D14 retention . Tom Laufer explained their approach of looking at long-term metrics and working backward to identify leading indicators at earlier stages, like day one retention, while conducting sanity checks to ensure correlation . The attendee acknowledged the potential for survivorship bias in this approach. Tom Laufer clarified that their models account for natural propensity to retain by considering factors like initial user behavior and marketing campaign origins to differentiate correlation from causation.