How I work

C. User Research (UR), Analytics, Product Managers (PMs), & Cohort Models: Applying scientific rigor to creative decision making

As designers, we have a keen instinct for the domains of usability, accessibility, cognition, and user needs. This holds true in proportion to how consistently we predict and design into these domains. But, inevitably, what is intuitive to us may not be intuitive to the user. As critical as social media and internal team feedback are to my process, they aren’t the whole picture, and so I prefer to validate my assumptions with scientific rigor. For this, I look to professionally gathered and analyzed data, leaning on external UR, user behaviors constructed from Analytics data, and by working closely with PMs whom I’ve found to have the exceptional holistic understanding of the intersection of consumer behaviors and Business Initiatives (BI)

• Working with UR
  • As soon as UR has cycles I like to start validating the design and will continue to do so as long as they have time and I have legitimate inquiries. If they have capacity but I don’t have testable artifacts I’ll make some, targeting questions essential to the success of the product.
  • Collaborating on artifacts
    • I think this often starts well before there’s any work to do, partnering early and often on desirable processes practices
    • I find it’s helpful to have a few conversations with UR leading up to a scheduled playtest that may use my or my team’s prototypes
    • The goal of these conversations is to understand the workflow and specialization of the researcher
    • Some researchers are very comfortable with verbally walking the user through a half-baked design to get feedback, others prefer prototypes that closely reflect the finished product.
  • Targeting specific problems in the product and avoiding sprawl
    • My goal is to test something specific and verifiable. Often this takes the form of questions assessing the user’s ability to successfully complete prescribed tasks that are central to the product’s success
    • As an example, if I’m working on a inventory management system, I’ll create very specific hypotheses that are appropriate for the state of development; for instance inquiring whether the user is able to find a set of item types, based on specific cohort’s need profile, at a specific stage in their time in the product
    • As the feature progresses, those user tests may get more elaborate or their goals may change to reflect the current state of the design or implementation.
  • Anecdotally, there are some more effective and less effective methodologies
    • One that I especially like is Task Completion Rate testing
      • Assign a Task – Ask the user to preform tasks that reflect those that they are likely to perform in the wild. they don’t need to be 100% accurate, but must test the user’s ability to find specific parts of the UI
        • This can be anything that demonstrates functionality of the system.
          • If they can’t perform the task, but should be able to at this point, there are problems with the design or functionality
      • User Report – Ask the participant to report on how well the experience was, how easy or difficult it was
        • This isn’t JUST about ease of use, if the user had a great time and the experience was challenging and delightful that’s a different kind of win
      • Invite Suggestions – Ask the user what they would like to see differently in the system
        • How could it have been easier, more enjoyable, clearer, more accessible, more efficient, etc.?
      • Posit Solutions – Evaluative Concept Testing, presenting solutions to the user’s anticipated grievances
        • Show prototypes and mockups that solve anticipated user suggestions
        • How well do they reflect what they suggested during the inquire phase?
        • How closely do the product goals, as expressed through the prototypes, match the user’s desires and expectations?
  • I try to observe participant playtesting whenever possible.
    • Watching their inputs, hearing the nuance and inflection in their responses, seeing their emotions with my own eyes can yield subtle insights into how they may engage with the product in the wild.
  • Applying extremely high standards
    • Like coming up with authentic personas and user stories, defining credible success/failure standards is essential. One of the main issues I see with user research is that it’s way too forgiving of tepid approval.
    • As product designers in an incredibly competitive product space, we’re under an “A+ or fail” set of standards
      • If we’re trying to change user behaviors, breaking through their existing habits
      • This is incredibly difficult and requires offering something either sufficiently novel and exciting or exceptionally functional… or both
      • I believe it is very difficult to tell leadership their strategy will fail. But an A+ in consumer eyes is terribly challenging to achieve so I try to be brutally honest about the product when discussing its prospects, even if users lean positively in favor of it
• Working with Analytics
  • Paint an early picture of the feature
    • As soon as the design is approaching a semblance of coherence
    • This is when I or other critical members of the team are confident enough in our state to have a technical conversation about analytics
    • Partner with Engineering and Product to define key telemetry, event tracking, and behavioral instrumentation early in the lifecycle to measure feature adoption and retention loops.
      • This usually focuses on behaviors related to KPIs we’re trying to achieve
  • Getting engineers planning early
    • With the target metrics list I start working with engineers
    • This can range from front end devs, to lower level systems, all the way to service engineers
    • we prioritize and cost each one, discuss the performance and server cost impact
    • We backlog the work for triage against all the other features
  • Pair technical experts on analytics and dev teams
  • Making and maintaining useful dashboards

• Working with PMs
  • Understanding the decision-making structure
  • Supporting PMs with leadership
    • I like to provide artifacts to help them make their case
    • Provide anecdotal support and council
      • Behavioral modelling
      • Feature analysis
      • Business hypotheticals
  • Partnering with PMs to interpret data patterns and inform good direction
    • User behaviors
  • If they’re not permitted to help to drive product strategy it bodes poorly for the organization

• Working with Segmentation Data
  • I often see expert cohort modelling that takes self-reported aggregate data and paints a picture of demographics which hold the greatest potential for engagement.
  • These demographics often look good on paper, capturing what seem like actionable behavioral patterns. For example, a report will cite that “70% of respondents said X so X will replicate with consumers,” while failing to address the specific conditions under which X occurs in the real world.
  • The issue is that these studies often describe a static user archetype, while ignoring human behavior over time and how patterns form, are carried out, and evaporate. They often fail to recognize that the target user ebbs and flows into particular states or that the target behavior only emerges over spans of time due to consistently suitable conditions.
  • A great example is the goal to attract social players, that often fails to acknowledge that social interactions are much more nuanced, occur more organically, than that goal might suggest.
  • To be fair, the social goal makes a lot of sense. Respondents often do report that their reason for not using the product is because, “their friends don’t play.” So we just gotta get their friends to play, right? Maybe, but what if their friends also say that they don’t play because their friends don’t play. It’s an Ouroboros of social attachment, a recursion with no base state. We just got to get their friends to play in order to get their friends to play so that their friends will play…
  • To me, these goals can be a little like aliens observing human behavior saying, “we need more human meat for the great harvest so we need them to procreate more! I see that these humans procreate on beds. If we put more beds into places where there are humans (the mall, restaurants, the DMV, places of worship) they will procreate more and make more meat for the great harvest!”
  • In reality, a system may be designed to facilitate social interactions based on conditions where social interactions take place, but if those users aren’t comfortable with those interactions, they won’t participate. Many years of study of social psychology and building features in the hope to cultivate user ties has taught me that effective relationship building in virtual spaces is a byproduct of creating the right conditions and providing the right tools.
  • The goal of saying “we need more social users so lets build social systems” is kind of like saying I need money to be rich. Social tie wealth in a product, like personal wealth, is the result of a series of seemingly unrelated systems.
  • While I believe that users are pretty sophisticated at self reporting the types of products and systems they like or would want, they may not be the best at the kind of deep introspection of their moment-to-moment motivations required to definitively tell us what less obvious solutions would lead to increased social interaction.

A closing note on Data collection & Analysis

Relying on a single data source creates critical blind spots in product development. To build accurate behavioral models and mitigate design risk, data must be triangulated across three distinct lenses—moving from sentiment on social media, to internal building internal consensus, and finally to empirical validation through both well-established and innovative data collection and analysis processes.

• Wave 1: Social Media & Community (The Canary in the Mine)
  • Provides early qualitative signals and leading indicators of player/user sentiment. Social feedback highlights real-world user friction, expectations, and emotional resonance 6–12 months before standard survey metrics (like NPS) reflect the shift.
• Wave 2: Internal UR & Team Playtesting (The Grounded Experts)
  • Leverages the collective intuition of cross-functional development teams. When developers actively engage with the product, they identify design flaws and execution gaps early, serving as an essential check against isolated executive vision.
• Wave 3: External UR, Analytics & Behavioral Cohorts (Scientific Rigor)
  • Validates assumptions using empirical, task-based usability testing (such as the TRIP framework) and telemetry instrumentation. Working alongside Product Managers and Engineers, this phase translates qualitative sentiment into measurable KPIs, verifying actual user mechanics over static self-reported data.