AI-Assisted Shopping
Evaluated an AI-guided purchase experience through moderated user testing
Role: Lead UX Researcher (study design, moderation, synthesis, reporting)
Method: Moderated prototype testing
Deliverables: Executive summary, detailed findings, prioritized design recommendations
Contents:
Representative AI Illustration - Not real design
Overview
I led six moderated user testing sessions with prospective and new customers to evaluate an AI-assisted shopping experience concept. Five themes emerged:
-
The AI-guided flow made shopping feel easier and less overwhelming for most participants.
-
Participants trusted the assistant’s guidance but still validated their decisions before committing.
-
Transparency, control, and clear value were critical to trust when personal data was used.
-
Context retention and saved progress made the experience feel seamless, but customers needed reassurance that progress would persist.
-
Pricing felt transparent, but key cost factors needed to surface earlier in the flow.
The findings helped the project team and leadership validate the strategic direction of the concept and informed subsequent design iterations to improve messaging around personal data usage, plan comparison, cost visibility, flexibility, and continuity across the journey.
Context & problem
As AI was becoming increasingly common in consumer experiences, AT&T explored an AI-assisted, personalized shopping journey designed to help customers navigate complex purchase decisions and differentiate its digital shopping experience within the telecom market.
While AI assistants and personalization can simplify the shopping journey, these capabilities can also introduce concerns around customer trust. When the system appears to know information about a customer or offers personalized recommendations, customers may question how their information is being used and whether the technology is acting in their best interest.
At the time of the study, the guided experience was an early-stage concept. There was limited understanding of how customers experienced the end-to-end flow, whether it reduced effort, and whether personalization was perceived as helpful rather than intrusive. The team required evidence to inform decisions about what to build, refine, or reconsider before further investment.
The study was organized around five research questions:
-
Ease: Does the experience make the shopping and purchase process simpler and easier?
-
Decision confidence: Does it improve customers’ confidence in their decision-making?
-
Trust: Do customers trust the experience and its recommendations?
-
Pricing clarity: Is pricing presented clearly throughout the flow?
-
Purchase intent: Does the experience increase purchase intent?
Research roadmap
This study was part of a broader, multi-stage research effort to shape the re-designed purchase experience. The first step was to synthesize known customer needs and frictions across the sales journey to help frame the opportunity space and clarify where the current experience was falling short.
The next phase was a large-sample concept test with nearly 1,000 participants, which evaluated customer reactions to multiple future-state AI concepts. That survey revealed broad interest in AI-enabled support, while also surfacing an important theme: customers valued the convenience and guidance, but their receptiveness declined as the experience became more autonomous, and they lost control.
This moderated study began the transition from broad concept testing toward testing the design itself. We were still largely focused on how the concept resonated with customers, but we were now also concerned with specific design elements and content within the flow. However, given the early nature of the prototype and its minimal interactivity, we were not yet at the stage of evaluating pure usability.
Approach
Planning & alignment
Because the project involved multiple teams, defining the scope of the study required alignment across business, product, and design partners. I drafted an initial research plan and circulated it for review, then finalized the discussion guide based on the feedback I received. This ensured the questions we set out to answer informed key decisions the team needed to make.
Method
I conducted six moderated customer interviews with a mix of prospective customers (actively shopping for new telecom services) and customers who recently signed up. Prospective customers were especially important because they were closer to the decision-making mindset of the target experience and more likely to have relevant questions, comparisons, and concerns top of mind. Recent customers who had signed up within the past three months offered a complementary perspective because they could compare their recent AT&T shopping experience with the new concept. I also recruited across age groups and levels of openness to AI, screening out participants who were strongly opposed to AI in general, so the study could focus on how the concept performed among customers who might realistically consider using it.
I chose a small-scale qualitative approach because the concept was still in a formative stage and required depth of understanding rather than statistical validation. The sessions were designed to surface nuanced reactions to the guided experience, including how customers interpreted personal data requests, what created discomfort, and what might make the experience feel more useful and trustworthy. Rather than measuring success at scale, the goal was to identify what was working, what was not working, and how the design could be improved to provide greater value to customers.
Moderation Strategies
During moderated user testing, surface-level reactions are sometimes ambiguous or grounded in inaccurate assumptions about the concept being tested. Consequently, as a moderator, I engage in targeted probing and follow-up questions to elicit more concrete and valid data. Two examples from this study:
When one participant saw a discount automatically applied based on his occupation, he described it as a clear positive and raised no concerns. Rather than accept that reaction at face value, I asked how he thought the system had identified his profession, and he explained that he assumed he had been signed in. When I clarified that he had not been, his response shifted sharply, and he said the experience would feel invasive enough that he might abandon it. Probing further on what would ease that discomfort revealed that acceptability depended on the source of the information, not transparency alone.
Before beginning each prototype walkthrough, I asked participants to do their best to imagine interacting with the experience in real life rather than simply going through the motions, and made clear that if they felt they would leave the page or abandon the site entirely, that was exactly what I wanted to hear. Even with that framing, I could still sense certain times when participants were being overly agreeable or advancing through the flow without too much thought. In those moments, it was worth nudging participants to think more about their realistic behavior. For example, when one participant said she was sold on the recommended plan and ready to add it to her cart, I paused before moving forward and asked whether she would truly be ready to commit at that point. That prompt led her to reconsider and note that she would first open the plan editor to test other configurations before committing, revealing a validation step she would have skipped due to the artificial setting of a usability session.
Collaboration during testing
I encouraged stakeholders to observe the sessions and scheduled optional debrief discussions immediately following each one, giving the team a consistent opportunity to share initial reactions while the session was still fresh. These conversations surfaced questions and interpretations early, which strengthened the eventual synthesis and kept the team engaged with the research as it unfolded rather than waiting for a final readout.
I also monitored messages from observing team members during the sessions themselves, allowing me to adjust my questioning on the fly (when feasible) when there were specific areas they wanted explored further. This made each session a shared resource rather than a fixed script, and meant the team could pursue emerging questions in real time without waiting for the next round of research.
The synthesis process
Using AI for assistance, with frequent verification of its output, I reviewed transcripts and session recordings and documented key observations for each participant at each stage of the journey. I then compared those observations across participants to identify where reactions converged or diverged.
From there, I grouped evidence around the primary research questions, looking for patterns related to effort, decision confidence, trust, pricing clarity, and purchase intent while also highlighting any observations that could inform design changes. As themes emerged, I captured relevant customer quotes to preserve participant voice, ground the findings in direct evidence, and help tell a clear data story.
I then organized the themes into two layers: key findings about the end-to-end experience and more specific insights tied to individual journey steps. This helped distinguish issues that reflected broader customer expectations from those that pointed to targeted design opportunities within the flow.
Key findings
The findings are organized around the major experience-level patterns that emerged during the study, showing where the AI-guided journey created value and where there were certain gaps.
1. AI-guided shopping made the experience feel easier and less overwhelming
Participants consistently described the experience as smooth, low-effort, and less overwhelming than shopping on their own. The assistant reduced cognitive load by taking on some of the research, comparison, and decision-making work that customers would otherwise have to manage themselves. For most participants, this added ease increased their reported likelihood to sign up with AT&T.
"The information is easy to understand, and I didn't feel overwhelmed with anything. It's just a walk in the park, as far as the whole thing." (P1)
"I feel like it was a little bit easier than trying to search for plans and make a decision on my own, because it felt like I put in what I want, and it helps me decide based on what's important to me." (P3)
2. Participants trusted the AI assistant’s guidance, but still validated decisions before committing
Participants generally found the AI assistant trustworthy because its tone, guidance, and personalization made the experience feel helpful rather than sales-driven. Trust increased when recommendations were clearly explained and connected to customer needs. However, participants still treated the assistant as a decision-support tool rather than a replacement for their own judgment, and most wanted to compare options, do additional research, or get human confirmation before committing.
"I trust [the AI assistant] more than I trust a salesperson who may be on commission. I feel like [the AI assistant] is there to help me, not sell me something." (P3)
"From here, I would really do my own research. I would worry that there was some kind of deal struck to make this the recommended device. So, I would look at the iPhone 17 on my own. But, I do appreciate that it's recommending this based on the options that I wanted." (P2)
3. Transparency, control, and clear value were critical when personal data was used
Participants consistently welcomed the value and convenience created by personalized content. However, those same moments created hesitation when customers did not understand how information had been obtained or what the system was accessing. Trust increased when the benefit was clear and customers had visibility and control over how their data was used, especially through opt-in consent.
"It's a double-edged sword… I'd be happy if I got the discount. I would feel like it's a little creepy because I didn't tell it that… but at the same time, I would be appreciative… For me, the fact that I know I'm eligible for a discount would be higher than the creepiness that I might feel." (P5)
"If there was a prompt before it like, 'hey, can we access your data to potentially find the best deals for you?'… If it's doing that without consent, then it gets into some muddy waters." (P2)
4. Context retention made the experience feel seamless, but users needed reassurance that progress would persist
Participants valued being able to move between questions, pages, and sessions without losing the thread of the shopping journey. When the assistant preserved context, the experience felt seamless and forgiving, allowing customers to explore independently while still returning to guided support. However, several participants initially worried they might lose progress if they left the flow, which meant continuity needed to be communicated clearly to deliver its full value.
"I like that it's a drop-down without losing the plan that you had. It kept me on my current page. That would be frustrating if it went back and forth." (P2)
"I would hope I don't have to start over, but in my experience, it feels like every website's different. I feel like maybe it would just go away, and I'd have to start over." (P3)
5. Pricing felt transparent, but key cost factors needed to surface earlier
Participants generally found pricing clear and easy to follow, which helped the experience feel more upfront than typical telecom shopping journeys. At the same time, certain decision-driving pricing factors surfaced too late, including trade-in eligibility/value and multi-line pricing. Because these costs could directly shape whether and how customers moved forward, they needed to appear earlier in the flow.
"As for the pricing, I feel like that's really clear, which is nice, because most times it's not as transparent. I don't think I'd necessarily have any questions on pricing." (P6)
"I think it should be early on, because I actually made my decision based on that." (P5, on trade-in)
Findings by journey stage
The full deliverable paired these experience-level findings with a more detailed layer mapped to each step of the journey. The map below summarizes those stage-level findings and the design opportunities they pointed to.
.png)
Recommendations
I translated the findings into a prioritized list of recommendations focused on improving trust, decision confidence, and continuity across the shopping journey.
-
Provide transparency and opt-in control over personal data use to support trust, particularly as data usage becomes more overt.
-
Treat plan comparison as a primary path to decision confidence, with a clear, visually scannable side-by-side view that AI assistance complements rather than replaces.
-
Surface key cost factors earlier, before commitment, including trade-in eligibility/value and multi-line pricing, as these directly influenced participants’ decisions.
-
Clearly communicate the affordance of continuity and saved progress, reassuring customers that their place in the journey will persist and keeping the assistant accessible as they move across pages or return later.
-
Support flexible pathways at every step, including a manual option for adding a second line, so customers can accomplish tasks on their own terms rather than depending solely on the assistant.
Additional recommendations addressed clearer feature labeling, broader signaling of the assistant’s capabilities, stronger rationale for AI recommendations, and clearer value communication for AT&T Guarantee messaging.
Impact
The research was shared with leadership and cross-functional stakeholders, where it helped the team evaluate the strategic direction of the AI-guided purchase experience and the case for continued investment. The study clarified where the experience showed promise and what conditions would need to be met for the experience to earn trust.
The findings also informed subsequent design iterations. The prioritized recommendations guided refinements to personal data messaging, plan comparison, cost visibility, flexibility, and continuity across the shopping journey. As a result, the research operated at two levels: helping leadership assess the opportunity and providing the product and design teams with concrete guidance for improving the experience.
Reflection
To extend this work, I would pair the formative findings with a larger-sample or more behaviorally oriented follow-up. Participants’ reported likelihood to sign up offered useful directional signal, but I treated it cautiously: small, moderated studies are well suited to understanding perceptions, barriers, and decision criteria, but stated intent is an unreliable predictor of actual behavior, and people are often poor at forecasting their own future actions. Measuring purchase intent more credibly would call for a larger sample and, ideally, behavioral rather than purely attitudinal measures.
The project also left me thinking more carefully about transparency as something more situational than it first appears. Participants consistently said they wanted visibility and control when personal data was used, but stated preferences do not always translate into the optimal experience or business outcome. In some contexts, surfacing every personalization mechanism could add friction or draw attention to data use that customers would otherwise experience as seamless and helpful. The more useful takeaway was not to be maximally transparent at all times, but to calibrate transparency to the moment, being most explicit when data use felt unexpected, sensitive, or tied to a consequential decision.
More than any single finding, the project shaped how I approach research on AI-enabled experiences. Evaluating them well means going beyond usability to examine whether the experience earns trust, where that trust breaks down, and what conditions help customers feel informed and in control, while staying honest about what a given method can and cannot tell us.