Back
Mitigating Challenges of Feedback Collection for Alexa's Proactive Experiences
Improving Hunches, a feature that allows Alexa to take proactive actions based on user behaviour, such as adjusting the thermostat or locking doors.
Tools
Figma, Miro
Timeline
January to April 2024
My Role
UX Research, Design
Team
4 UX Researchers & Designers, 1 UX Strategist, 2 UXD
Challenge
Hunches can often leave users confused or frustrated due to a lack of awareness and unintuitive feedback processes.
The research identified key issues
Low engagement, Hidden flows & feedback options, Lack of user notifications.
Goal
To design an effective feedback mechanism, aiming to align these actions more closely with user preferences and improve overall accuracy.
This involves understanding
When and how users prefer to give feedback and ensuring they are informed about the proactive actions taken.
View Solution
Jump to High-fidelity Prototypes, Final User Flow Details
Presentation Deck
Look at the Sponsor Presentation by the Team
Why do we need feedback for such proactive experiences?
Let's say you are working till late night
Suddenly, Alexa turns off the light at 10 PM
You get confused and annoyed
Incorrect Action
Alexa needs more feedback to improve
Feedback
Fewer Incorrect Actions
Improve User's Trust
Based on your routine, Alexa assumed you went to bed and forgot to turn off the light. Therefore, she intervened by turning off the light.
Current mediums leveraged by Alexa to collect feedback
Alexa Mobile App
Echo Show Device
Conversational / Voice Interface
Hunches Feature
Why is the current feedback mechanism not working?
Users give feedback only when things go wrong
They love proactive actions, but hate proactive feedbacks
Users are largely unaware of incorrect proactive actions
Visibility of Hunches on Alexa App and Echo Device is poor
Location and Context of feedback notifications are random
Unaware how Hunches work & why their feedback matters
How did we reach here?
It was a lot of going back and forth - A complete Non-Linear Design Process.
Through our initial discussions trying to frame the project, we had identified probable domain knowledge areas to gather information on.
Disciplinary Knowledge
Cultural & Contextual Awareness
Conversational Feeback
Consumer Insights
Behavioral Psychology
This helped us shape our approach and focus on closely relevant areas, guiding our decision-making.
How do users resonate with and perceive proactive actions?
What are the optimal impacts & effectiveness of proactive content?
What does it mean for the user to provide feedback?
What methods of feedback feel natural to the users?
We had several questions to begin with.
Direct Competitors
Indirect Competitors
We coded (tagging) more data from competitor analysis on Conversations, Proactive Features, Playfulness and Context Awareness.
This resulted in a richer understanding of the temporal content of feedback and user agency.
It also gave us insights how visual feedback and push notifications can interplay with contextual awareness of the user.
Data coding helped us map and link how user agency and context play a critical role in feedback collection.
Scouting Forums
Digital Ethnography
Studying people’s interactions with Alexa's voice commands and their reactions to hunches.
More than 80% users on forums such as Reddit, Discord & Twitter (X) found feedback method intrusive and random. Many questioned if it makes any difference?
To verify our findings, we decided to do Contextual Inquiries and Diary Studies.
But our attempt failed. Why?
Meanwhile, our findings from Competitor Analysis over factors such as Conversations, Playfulness, Context Awareness and Proactive Features gave us clarity in our approach.
Many users did not understand if the proactive actions were triggered by hunches and considered it as a bug.
Tends to disregard user's context
& environment resulting in being irrelevant and disruptive, especially with repetitive use of phrases.
Unaware of triggered actions unless explicitly accessing activity section of the app where they can provide feedback
User Agency
User’s Lack of Knowledge
Timing
Personalisation
Context Awareness
User Agency
Feedback notifications are inconsistent and triggered only when Alexa sees a possibility of an error.
In a nutshell it's unpredictable when the user will interact with the feedback mechanism.
Timeline limitations & logistical challenges made it difficult to conduct these studies.
Hence, we shifted to a more practical approach.
User Interviews
Co-design Workshops
Conducted 10 Interviews and 2 Codesign workshop with over 6 participants. We organized the evidence we collected into groups of possible insights that could be drawn from users. These insights boiled down into the following relevant pain points. We targeted these pain points when we did initial ideations.
User Interviews & Codesign
Complexity in Providing Feedback
Users expressed confusion and frustration when unfamiliar with hunches, leading to a lack of feedback. Additionally, it is hidden 4 layers inside from the home screen of the alexa app.
90 %
Preference for Passive Feedback
Users prefer non-disruptive feedback methods, such as notifications, over voice prompts. This highlights the need for semi-proactive feedback requests that minimize disruption.
All
Contextual Sensitivity in Feedback Requests
Users do not prefer to provide feedback when they are busy but are willing to do so if the matter is urgent - implying the need of feedback prioritisation into low/high priority scenarios.
75 %
Our approach to handling hunches centered on utilizing conversational interfaces to make the process less intrusive while still encouraging user feedback.
Our primary challenge was devising solutions that neither irritate users nor overwhelm them with information.
While Ideations happened simultaneously with research, we had a critical pivot in our approach after Sponsor meeting and Critique session.
Focus
Device level (high-low) priority categorization.
Increasing visibility in existing application.
Providing Impact of feedback to make it meaningful.
Ideation Phase 1
In addition to the focus points, we explored various possibilities in methods of asking feedback across channels in Alexa ecosystem. The aim was to leverage different channels to craft a cohesive solution and to understand if the method of asking for feedback would differ across scenarios and devices.
Phase 2 of Ideations and how we reached to the Final Solution
Increasing visibility and awareness, easing the user flow
Tagging to increase user context
Adding steps to feedback and asking for specifics
Notifying user of triggered hunches
Current Screen
Widgets can be leveraged as a medium to keep users notified about hunches and encourage them to provide feedback with minimal actions.
Asking users more context about the scenario giving quick common options to select.
Color signifiers on Alexa Device to notify user of a triggered action.
High Stake Notifications can be instantly notified and users can tap to expand and give feedback without having to open the app.
Stitching Device-level Scenarios into a Cohesive Experience
Methods used: Low-Fi Wireframing, Hi-Fidelity Prototyping, Cross team design critiques
Design Features + Prototype
The user would then be able to access the Hunches Activity Page which would give more information about the detailed Hunches Activities by tapping on the Widget.
When Alexa’s action goes wrong, users' first instinct is to check the Alexa Mobile App’s Activity Screen.
Widget reduce the number of clicks to get to Hunches Activity Screen and provides updates semi-proactively.
By leveraging the Echo Show's always-on display, we can grab the user's attention to notify them of proactive actions triggered by Alexa without being intrusive.
Pulsating Motion widget which would signify the latest Hunches triggered by Alexa
Insight 1
Lack of User Awareness around Hunches
Action
Semi-Proactive Notifications
Insight 2
Intrusive notifications are preferred during high-stakes situations
Action
Discarding On/Off Hunches and introducing device level alerts
Although the latter can be more intrusive, user interviews showed a preference for intrusive notifications in high-stake situations.
Insight 2
Hunches' activities hidden deep in app navigation
Action
Display latest hunches upfront on the home screen
A new "Hunches" section now appears on the home screen to alert users of actions initiated by Alexa
A new tag introduced in the device tile will inform users of proactive actions performed on it
Insight 2
Transparent feedback impact increases user engagement
Action
Showcase Hunch Model Improvement overtime based on user feeback
Assumption would expand and ask the user for more context.
Research insights suggested changing technical terminologies (eg. Hunches > Assumptions) makes it more user friendly.
Future Scope
User Testing & Iterate Iterate Iterate
If Alexa notices a recurring feedback for a particular device, it could ask the user to change it to a routine activity, which is an existing feature in the amazon alexa app. We want to develop this idea further and test it.
Explore Voice Interface & Integration in Routines
Our research shows voice interactions for hunches are slow and intrusive, especially for users with multiple devices, so while we offer voice notifications during setup, further user testing is needed to decide their viability.
and that's me!
Reflections & Takeaways
Being comfortable with ambiguity
The project initiated with a vague problem statement with many possibilities. I learnt to navigate ambiguous spaces by leveraging design thinking and problem solving abilities by trusting the design process. I learnt the non-linear way of approaching problems.
To design Hybrid Interfaces
Designing a solution for a conversational interface does not necessarily imply utilising Voice UI. There are a number of ways to any problem can be approached by exploiting different channels and touchpoints.
Resume
Copyright © 2024 Shantanu Thorat
Made with love in Bloomington, USA