Usability Testing for New Product Launch 

@ Exabeam

Overview

This is an evaluative research initiative I led at Exabeam. Alert Triage was the product area focus. This was part of a design partner program that was launched to promote iterative research with our customers. The purpose of this program was to design products with our customers, including them early in the design process to get their feedback on our product direction before it became generally available.

Understanding a persona we’ve never serviced before

Within a security team, there are three tiers of analysts. Exabeam’s product suite supported Tier 2 and Tier 3 analysts, but was lacking a tool that helped Tier 1 analysts triage thousands of alerts they received on a daily basis. In order to build a product that catered toward Tier 1 analysts, we needed to

  1. Understand their workflow and how they triage alerts currently

  2. Learn what information is needed to take action on an alert

Research Planning Obstacles

Recruiting from a small pool of users

In the cybersecurity industry, the pool of users to select from is small due to the complex nature of the processes and products used to tackle cyber threats. The individuals we needed to talk to required a specific skill set i.e. knowing how to triage security alerts from different security vendors.

Working with sensitive and real world data

Tier 1 analysts are trained to look at security alerts that are specific to their organization. Having an environment that simulated data that was relevant to them was crucial in understanding what information is needed to take action on an alert.

Following an aggressive timeline

The scrum team was working on an aggressive timeline, which meant we needed to get valuable insights within a short amount of time.

How are we going to overcome those obstacles? Partner with customers

Our customers had analysts that triaged alerts, environments that had real world data that we could use, and regularly communicated with Exabeam’s Customer Success(CS) team which meant they were easy to get a hold of and establish a working partnership with.

I defined what it meant to be a design partner and the benefits that came along with it. After solidifying the requirements, I met with our CS team to communicate what we were trying to achieve and asked them for help figuring out how to implement this program. We gathered a short list of customers that would be a good fit for this program(team size, availability, willingness to provide feedback and collaborate, have security alerts from different vendors, etc), started reaching out, and set up introductory meetings.

A majority of our customers were unfamiliar with UX research, so it was important that their first experience with our team went well. I met with the lead product designers and product manager to discuss what was needed in order for these meetings to be successful. During our planning meeting, I asked a few questions to make sure we were on the same page and set expectations on how meetings with partners were going to be run.

Prepping for design partner meetings

What questions do you have?

  • What is their current triage process like? 

  • Do we display the right information that helps analysts take action on an alert? 

  • Are they able to perform the three main actions: Dismiss, Escalate, Resolve? 

What are the goals? 

  • Ensure analysts have information needed to take action on an Alert

  • Ensure analysts know how to dismiss, escalate, or resolve an alert

  • Understand where the pain points are and address them in the next round of designs

What are we testing? 

  • Live environment that is pulling real data from customer databases.

Method

For the first meeting with our design partners, we started with a usability test followed by a System Usability Scale (SUS) survey. This would help us gauge the current usability of the product, while also giving our partners a taste of what UX research is and how they can help influence the design and implementation of the product.

Sample Agenda:

  1. Introductions / Setting Expectations

  2. Background questions

  3. Usability Test Tasks

    1. Example Task: Let’s imagine you need to dismiss an alert that does not need to be investigated further. Can you show me how you would do this?

  4. Wrap-up Questions

  5. Share SUS Survey

Findings

The final usability score was 79.7, which meant that the usability of the system was acceptable. However, after speaking with and observing our users, there were several improvements that needed to be made in order to help analysts quickly and efficiently take actions on the security alerts in a given alert channel.

The design had two levels of information which we referred to as L1 and L2.

In L1, the combination of not enough data plus the lack of context (no column headers) resulted in the inability for users to take action. We needed to learn more about what data needs to be displayed to make it easily scannable for analysts to quickly dismiss alerts. 

In L2, the parsed alert details weren’t enough for an analyst to confidently change the status of an alert. They needed to look at the raw log or pivot to other systems to get more context about the alert. More fields from the raw log needed to be shown in Alert Details [L2] to help analysts take action on an alert.

For more detailed information on the findings, you can contact me at cmvtran@gmail.com.

Research Impact

Product change

  • Increased usability score

    • This UX research led to insights that resulted in changes to the product. The changes made increased the usability score from 79.7(Acceptable) to 82.5 (Good).

  • Prompted more research

    • The research opened the door to more questions from product managers and designers which meant more sessions with design partners. As questions were answered, designers would make changes to the prototype and we would go back and test concepts with our partners.

Strategic Impact

  • Design Partner Program formalized and more research started for other projects

    • The success of this research activity paved the way for more UX research to be done with Alert Triage and on other products. The program was formalized and it was used for two other high visibility launches.

  • Visibility with leadership

    • UXR became a topic of discussion at product and engineering leadership meetings, which meant more visibility and support from leadership.

Customer Appreciation

A quote from a customer after our session was over:

Thank you all for the time and I do, I'll say again, I really liked the tool and I like where it's going. And I appreciate the fact that you guys are doing the sessions…and taking feedback and improving based on recommendations from the you know, the whole user base.  That's really important. So thank you.

Learnings

  • Working as a UX Research team of one requires focus, organization, and support from the team

    • Writing a detailed research plan and setting deadlines for each action item is important to stay on track

    • Setting expectations with the team on what you can/cannot do will avoid surprises during the process

  • Stakeholder buy-in and collaboration is important

    • 1-1 meetings with designers and PMs are needed to understand the project. It also helps with the research planning process (questions to ask, understanding the domain, recruiting participants, scheduling)

    • Seeing in real time how their product is being used will make them more inclined to act upon what users are saying

  • Practice makes perfect

    • Doing a dry-run of the script with your team will help you figure out what is working and what needs improvement