Neural Notes
with Viral Communications Group at the MIT Media Lab
Role & Team
Supervised by Media Lab PhD candidate Mike H. Jiang, I’m the sole UX designer on the team.
A web-based music application aims to incentive creativity through Jazz trading with machine learning algorithms.
Timeline
January - May 2023
Tools
Figma, Miro, Adobe illustrator
Context
Developed at the Viral Communications Group at MIT Media Lab, Neural Notes is an interactive web application that aims to help its users, who are Jazz musicians, to become better jazz improvisers. The idea is that the user and the machine take turn in playing musical phrases and while doing so, help the user to become more creative. The project has multiple legs, including Machine Learning research and Human-Computer Interaction research. I’m working on the human-computer interaction leg of the project.
Before delving into details of the problem, I first took time to understand some musical terms that relate to Jazz and improvisation:
Improvisation: the activity of making or doing something not planned beforehand. In music, it means spontaneous performance without scripted preparation. It’s also a major part of Jazz music.
Jazz Trading: a group of musicians take turns improvising solos that and respond and interact with each other’s music, showcasing their individual creativity and musicality.
Design Process
FOUNDATIONAL RESEARCH
The Challenge
At the start of this project, we spoke to Jazz musicians who told about their struggle practicing jazz improvisations. As improvisations in jazz always happen in a group setting, it requires scheduling between multiple people to find a right time to practice. It’s not easy to find partners to practice improvisation with.
To help address this common challenge that jazz musicians face, I worked with my team to identify a better solution.
What I learnt…
Market Analysis
Has any existing software already addressed the challenges of practicing improvisation? Is there a demand for a new application in this area? With the user challenge in mind, I explored some existing products, learning what made them successfully along with any missed opportunities presented.
Learnings:
Machine Learning can be used to improvise with users real-time.
Essential toolbar features for jazz improvisation.
Opportunities:
Specialization: An app solely dedicated to improvisation practice.
Customizable: Enable users to customize the configuration of their AI improvisation partner.
Intuitive, Visually Appealing Design.
Impro-Visor
Jazz improvisation/ machine learning research project from CMU.
Learnings:
Minimal, intuitive drag-and-drop interaction and design system.
Easy to navigate between pages.
Text explanation accompanies more complex icons.
Opportunities:
A need for Improvisation Practice features.
Feedback: utilize machine learning to give personalized feedbacks based on past improvisations.
MuseScore 4
Free, open source music notation software.
DEFINE
User Interview
Understanding there is a gap in the current market, I conducted a detailed user interview session to understand the specific needs of a Jazz improvisation application with experienced Jazz musicians. Additionally, as a HCI research project, I also talked with HCI researchers to understand their need on the other side of the app.
IDEATE
I’ve found user flow mapping a great tool to convert ideas into reality. While creating user flows, I put myself into user’s shoes and wrap my head around what would be the most intuitive user experience.
I also made sure to communicate with project lead and users consistently to validate that the flow makes sense.
“Pause doesn’t make sense during improvisation. Everything is completed in one round during improvising. ” ——Feedback from a Jazz Musician
“Instead of automatically saving a recording, user needs to provide clear decision to “save” or “delete” a recording session for researchers to analyze clear user data.” —— Feedback from a HCI researcher on the team.
Design 1
Improvise Page
How might we provide Jazz musicians with an intuitive, customizable improvisation practice platform with access to feedbacks of past improvisations?
How might we provide researchers with a data gathering tool to collect clear and precise user input?
The Solution
An Improvise page, where users improvise Jazz trading with customizable machine learning algorithms.
A recordings page, where users can revisit their past improvisations, and duplicate the same setting to create new improvisations.
An analytics page, where user can gain feedback about their improvisation style based on their past improvisations, and be advised on ways to be more creative. (WIP)
User Flow
PROTOTYPE
User Feedbacks and Improvements
Ideating the record button: I first made sure it’s feasible to have the button change into different stages, then conducted A/B testing with the engineers to pick out the best design.
Design 1
Low-fidelity prototypes
Low fidelity prototype is the start of my visual problem solving journey. While building the structure of the application through wireframing, I also made sure to be empathetic to the engineers. I prepared multiple version of the designs to discuss their feasibility, and we would set strategic goals of the features and the designs to make sure the MVP is first met.
MVP Design
Even though Design 1 is a more creative: it has previews of the music sheet with the information about the past recording sessions, we decided to choose the table design as the MVP design. It’s simpler and easier to implement, but still shows the essential information —— the priority of the project is for it to be functional first.
While proceeding to the high-fidelity design, I iterated and tested 2 more times based on feedback.
Design 2
I conducted A/B testing with the engineers on the visual design of the app. 5/5 users picked design 2, as it’s more accessible to read and catches the eye more.
after
TEST & ITERATIONS
Jazz Musicians
Frustrations
Struggles with finding partners to practice Jazz trading improvisations with.
Feels he’s been following similar improvisation patterns.
Lack of feedbacks about his improvisations.
HCI Researchers.
Wants
An application that can record clear data about user decisions to conduct HCI research with.
before
The feedback from an experienced jazz musician during a usability testing session is that, when musicians improvise, they don’t need the playback feature until the recording session ends.
So in my final prototype, I removed the playback feature when the user is improvising, and only show it when the improvisation is finished to re-listen to the recording and decide to keep it or delete it. It once again shows the importance of being consistent with user flows.
Final Prototype
Intuitive, uninterrupted improvise experience
The black and white color scheme allows users to focus on the music sheets.
Minimal buttons enables an uninterrupted, fast paced improvise experience.
Playback functions shows up after recording session ends.
A recording library with thumbnails of past improvisation sessions
Preview and overview the past projects at a glance.
Instant access to new improvise sessions.
Duplicate the same setting to create a new improvise session.
Your Recordings Page
Home Page
Reflections
Working on Neural Notes at MIT Media Lab was a great learning experience for me because it’s my first time being the sole designer on the team. I really enjoyed the cross-functional, collaborative experience working in a small team that mimics a startup environment, and the amount of impact I can make: I converted an idea into reality and made sure the application is user-centric in every step of the design.
This project also let me peek into the field of HCI research, which has been my academic interest for future studies. I learnt how to design for both the end-users and for research, as one of the challenging but also fun parts of the project is having 2 distinct user groups.
I would continue my work for Neural Notes in these areas:
Designing an analytics page.
Coding with p5.js to creative interactive patterns that response to user’s music input.
If I have time & there’s need, I’d also implement the front end of the application using React.js
This project has been a very fun and creative experience. Thanks for taking the take to read about my work!
