OverviewSolutionResearchDesignResearch IIIdeationDesign IITestingDesign 3Testing

Accelerating data exploration through automation

Team & Role

I led this project under the mentorship of the Data Science group director at Nokia Bell Labs

What I did

Researched, designed and developed a prototype for data exploration in an autoML tool.

What I delivered

  • High fidelity wireframes
  • User feedback from 10 user interviews, 13 usability tests
  • Market Research
  • 3 Journey maps
  • Framework built on React.js and R
  • Product Branding Guide

Project Duration

June 2022 - May 2023

Note: Brand naming and design were done as part of a class project.

Problem space

About 50-80% of a data scientists time is spent on making sense out of large datasets

Exploratory data analysis is a critical process in data science, as it lays the foundation for insights and models to be developed. However, this process can be time-consuming and cognitively overwhelming, with most data scientists spending more than half of their time studying datasets with 100+ columns.

While the past few years have seen a large rise in the number of MLOps and AutoML tools that look to automate the ML pipeline, data analysis continues to be a manual process.

Breaking it down...

Phase 1 (3 months): How can I design a structured framework to reduce the cognitive load during data sensemaking?


Phase 2 ( 6 months): Most automated solutions are often seen by experts as "blackboxes" that offer no transparency and leave users out of the loop. How can I integrate flows to boost interactivity, trust and control? 

Process

How can a tool support and augment the user while still ensuring they are in control?

Exploratory analysis can be overwhelming when faced with a large, unfamiliar dataset. How can its UX help ease users into a dataset, and tell a story?
‍
People are skeptical when any automation is involved. At what points can we improve transparency to boost user trust?

Solution

A trustworthy and layered data exploration framework that accelerates discovery through automation.
I designed a UX to ease users into a tabular dataset using the martini-glass storytelling approach.
An information architecture that allows users to explore data at different granularities using relevant automated insights for each level.
Overview visualizations that give users potential entrypoints into large, complex datasets and interactions to help users narrow down on interesting columns
Transparent and explainable automated insights, to help users understand system actions under-the-hood.
3 stages of verification that boosted user trust in automated insights by 80%.
Final layer to allow users to conduct their own exploration and generate visualizations for starred columns.

Problem space

About 50-80% of a data scientists time is spent on making sense out of large datasets

Despite the increasing trend in automating different parts of the DS lifecyle, and increasing usage patterns of autoML tools such as VertexAI and AzureML, data exploration continues to be a painstakingly time consuming and manual process. Due to the amount of creativity required in this phase, they must be carefully designed to place the control in the hands of the human. At what points can automation be introduced to augment human creativity and curiousity? 

User Research I

‍

Research goals

Understand how data scientists would explore an unfamiliar tabular dataset for a classification task and their experience with existing autoML tools.

‍

‍

My research goals of Round 1 were to understand how users would explore an unfamiliar tabular dataset for a classification task and their experience with existing autoML tools. To explore this, I conducted 45 minute semi-structured with 6 data scientists within a research team.

‍

Key Finding #1

Users had different perspectives on data exploration

Some users relied heavily on modeling to understand the data, and were looking for tools that would give them a stronger grasp.

‍
Key Finding #2

Users faced a high cognitive load while studying datasets

Users talked about feeling lost while studying large datasets, not knowing what to do with the data, and the danger of overlooking patterns.


Key Finding #3

Users felt a lack of control in existing autoML tools

Users talked about existing autoML tools being a “no-brainer” and “blackboxes”.

‍

‍

Finding #1

Users faced a high amount of cognitive load while studying data.

Design implication: Support users' exploratory workflows through better structure and guidance.
Finding #2

This led to some users brute forcing models to better understand their data.

Design implication: Encourage users to be curious and explore by highlighting strange things in a dataset.
Finding #3

All users agreeed that data quality was key to their sensemaking.

Design implication: Visualize the quality of a dataset to accelerate data analysis and sensemaking
Finding #4

Users were reluctant to shift to newer tools.

Design implication: Reduce the learning curve involved by designing flows that feel natural and familiar to users.

Design Goals

Design

EDA is deeply layered and involves inspecting the data at different granularities. What insights can we automate to help accelerate this process? 
My solution was to build a user experience around zooming in and zooming out of a dataset.
I structured the data sensemaking process and insights into 3 levels of granularity.
My goal was to design a layout that allowed users to move between different granularities.

Concept 1: Quality Fingerprint

Understanding the quality of data was something crucial to users in the initial steps. The system could reduce the load placed on users by automatically calculating quality based on commonly used criteria, and visualizing its varying levels in a single view.

Concept 2: Quality Fingerprint with columns

While the previous concept prioritize visualizing the entire dataset in a single view, this concept focused on giving users contextual information about the columns through column names.

It also looked to provide users with a more natural transition into the second level of granularity by substituting colors with values when zooming in.

The previous two concepts used a lot of vertical space that did not convey any information. While it seemed more closer to how columns are viewed, it was important to prioritize efficiency over aesthetic considering the large amount of columns in a dataset.
A heatmap would allow users to view column names while accomadating more columns in a single view.
Users talked about the dangers of overlooking patterns while studying data. My solution was to provide users with a fingerprint of their dataset.
To ease the initial cognitive load placed on users, I proposed using level 1 of exploration to communicate global views of the dataset. By highlighting "strange things" in a dataset, these would also serve as more accessible entrypoints that encourage curiousity among users and guide a deeper exploration.

As an example, I chose to visualize the quality of a dataset for my prototype. Green indicated columns having minimal errors, while red highlighted columns showing various type of errors.
I used an interactive heatmap to help users visualize the quality of their data.
Level 2: Transitioning from system-driven exploration to user-driven exploration

Once users were familiarized with the data, the goal of level 2 & 3 was to pass the control onto the human and minimize system interventions.

Guiding users using automated insights

At this level, my goal was to help users navigate through the table using automated insights. The insights were designed to highlight strange things in a dataset that often tap into the curiousity of data scientists during exploration.

‍

Visualizing key relationships

Data scientists at the Bell Labs research group have created algorithms to extract relaionships from the data. This section houses them together, conveying important statistical features through network visualization diagrams.

‍

I designed system interventions and features that supported the data scientist in identifying interesting columns.
Level 3: Helping users inspect individual columns

Automated interactive visualizations

The final level of granularity helps data scientists inspect columns individually by evaluating its metadata and distribution of values across the column.

User Research II

I tested my prototype with users and dived deeper into key research areas.

‍

Research goals

Understand how quickly users were able to navigate through the dataset at different levels of detail and how well the system supported their data exploration workflow.

‍


The tasks given to users were: 

T1, Level 1: Gather an overview of the data using the visualization and insights in Level 1.
T2, Level 2: Use the automated insights in Level 2 to study specific columns
T3, Level 2: Understand relationships between columns through automated visualizations
T4, Level 2: Filter to generate automated insights for a subset of columns
T5, Level 3: Study columns individually and filter values using the interactive visualization

‍

Research goals

  1. Understand the influence of control on trust while working with automated tools
    ‍
  2. Understand the points at which users would want to interact with the system
    ‍
  3. Underestand how cognitive load can be reduce while working with large datasets
I created 3 journey maps to further identify design opportunities.

The scenarios I created journey maps for are: 
‍

  1. Manual analysis of data: Data scientist uses Python to build a machine learning model that predicts how capable each applicant is of paying back a loan. 
    ‍
  2. Analysis of data using Dataiku: Data scientist uses Dataiku to build a model for revenue forecasting. Dataiku is a leading competitor in the space of human-centered autoML that I identified during my market research.
    ‍
  3. Analysis of data using the augmented data science (ADS) tool developed by data scientists at Bell Labs: Data scientist uses ADS tool to build a machine learning model that predicts how capable each applicant is of paying back a loan.

‍

Key Findings

Finding #1

Users thought the tool gave them good starting points in a dataset

Finding #2

However, users had difficulties trusting the automated insights

Design implication: Integrate quantitative insights and eliminate qualitative language to boost user trust.
Finding #3

All users believed that their trust in the system overlapped with transparency & control

Design implication: Find the right balance between control and automation.
Finding #4

Some users wanted to see a different first view of the dataset

Design implication: Give users a more descriptive preview of the dataset before presenting information about its quality. Redesign the visualization to be scalable to 100+ columns.
Finding #5

Users saw the potential for more ways of narrowing down on columns

While the filter feature in level 2 was well recieved by all users, some users found it to be unituitive. Users seemed more comfortable filtering through columns at the overview level.
Design implication: Integrate a filter feature at the overview level, to further reduce the cognitive load placed on users during analysis.
Finding #6

Users did not find Level 3 too helpful to their analysis

Design implication: Redesign the final level to include more relationships exploration


Key Feedback

Design goals

Ideation

Some users preferred less colors and wanted to visualize all columns without scrolling while working with large data. How could I redesign it using Edward Tufte's visualization principles?
I combined visualization principles with user goals to further evaluate concepts for a global view.


Considering the high amount of cognitive load placed on users while studying tabular data, designing comprehensive visualizations to ease users into a dataset was one of my primary goals. To benchmark design concepts I created a list of inspired by 3 principles based on E. Tufte's guidelines and 3 from user feedback.

Initial concept: Encoding quality in an interactive heatmap

1. Encourage the eye to compare different pieces of data
2. Reveal the data at different levels of details
3. Maximize the data-ink ratio
4. Eliminate or reduce scrolling
5. Place higher importance on visualizing key distributions and column relationships
6. Allow users to filter, select, and drill down on subsets of columns

Initial concept

At this level, my goal was to help users navigate through the table using automated insights. The insights were designed to highlight strange things in a dataset that often tap into the curiousity of data scientists during exploration.

‍

Initial concept

At this level, my goal was to help users navigate through the table using automated insights. The insights were designed to highlight strange things in a dataset that often tap into the curiousity of data scientists during exploration.

‍

Initial concept

At this level, my goal was to help users navigate through the table using automated insights. The insights were designed to highlight strange things in a dataset that often tap into the curiousity of data scientists during exploration.

‍

Initial concept

At this level, my goal was to help users navigate through the table using automated insights. The insights were designed to highlight strange things in a dataset that often tap into the curiousity of data scientists during exploration.

‍

Final Concept: Connecting views using a brushing & linking interaction

1. Encourage the eye to compare different pieces of data
2. Reveal the data at different levels of details
3. Maximize the data-ink ratio
4. Eliminate or reduce scrolling
5. Place higher importance on visualizing key distributions and column relationships
6. Allow users to filter, select, and drill down on subsets of columns

An example of brushing & linking using D3.js

1. Encourage the eye to compare different pieces of data
2. Reveal the data at different levels of details
3. Maximize the data-ink ratio
4. Eliminate or reduce scrolling
5. Place higher importance on visualizing key distributions and column relationships
6. Allow users to filter, select, and drill down on subsets of columns
I sketched a storyboard and collected feedback from 4 users.

I had 20-minute short interview sessions with 2 visualization experts at Georgia Tech and 2 senior data scientists.

While I recieved positive feedback on the concepts, visualization experts suggested using lesser space for the main visualization and thinking about how users would decipher the column names when 50+ data points are involved.

Design

I brainstormed ideas from the lens of control, trust and cognitive load.

Having laid down the foundation of the tool in the previous design phase, the goal of this design phase was to further refine the system based on feedback from usability tests.

‍

Key Feature 1: I redesigned the layout to optimize it for interactivity, transparency and space.
Key Feature 2: I restructured the granularity levels to better match user expecations.

Users did not find the column-level visualizations too helpful in the previous design. Rather than viewing the distribution of individual columns, they would prefer to plot relationships between various columns.

Similarly, users mentioned they would prefer to see data that is more descriptive of the dataset before understanding its quality.

Key Feature 3: I allowed users alternate ways to explore data.

A key finding from usability tests was that users were not willing to trust the visualization without seeing quantitative information.

Will this design change be enough for users to trust the overview visualization?

Testing

‍

Research goals

  1. Evaluate user trust in system insights
  2. At what points do users demand more control?
  3. How do they use the visualization to explore the dataset?
I replicated flows from the journey maps to evaluate for control, trust and cognitive load.
T1: Preview the data
T2: Explore the layout of the system
T3: Analyze the quality of the columns
T4: Dive deeper into the poorer quality columns
T5: Create a subset of the columns
T6: Review automated insights
T7: Handle missing values in a column
T8: Review changes made by the system

Key Findings:

50% of the users found the overview visualization ambiguous.

2 users voiced their confusion on their first impression of the chart. However, once users spent a few minutes understanding the view, they were able to grasp the visualization.

Users comfort levels with the visualization possibly impacted their trust.

3 out of the 4 users seemed skeptical about the visualization. One user even counted the dots in the visualization to make sure all columns were correctly being represented!

Users continued to be skeptical about the automated quality score parameter.

Redesign

I updated the visualization to be more familiar with users.
I used user feedback to design a 3-step flow for validating system generated insights.

Step 1: How is the system generating a score?

4/4 users in the usability tests immediately wanted to know more about the insights were being calculated.

Step 2: Can I control the way in which it generates insights?

2/4 users in the usability tests wanted to be able to play around with the thresholds. This also aligned with a key finding from the exploratory interviews where 2/4 users mentioned that their trust in the system overlapped with how much control they had.

Step 3: Do the insights align with the data?

4/4 data scientists in the usability tests responded positively to histogram visualizations of each column. Allowing them to preview each column in this screen to validate errors noticed by the system, should allow data scientists to be more comfortable with the scores.

Testing

‍

Research goals

  1. Revaluate user trust in system insights
  2. Evaluate if users were satsified with the amount of control
  3. Evaluate how users would prefer to interact with the visualization for sensemaking

Key Findings

Trust scores in automated insights increased by 80%.

Users in this round of usability test followed a similar thought process while validating automated insights. Their trust in the system increased progressively across the 3 stages of validation. Unlike in the previous rounds of usability tests, users were comfortable with the insights generated by the system.

All users were comfortable with the overview visualization.

Unlike in the previous rounds of usability testing, where it took a while for users to grasp what the visualization was conveying, users in this round were immediately able to get an overview of the data. All users further interacted with the visualization by clicking on the bars to test their understanding.

There could be a possible overlap between novelty of visualization and user trust.

Across previous usability tests there were certain users who were skeptical about the insights as they were not familiar with the data visualization. However, 4/4 users were comfortable with representing columns using bar graphs and coincidentally had higher trust ratings in the system.