"
Cut policy research time by 50%
A policy research tool using NLP and ML technology
My role
Design lead
A policy research tool using NLP and ML technology
healthcare fraud, waste and abuse in 2016
Less than
is recovered
State Medicaid Program Integrity Units -
1. Limited resources
2. Lack of new technology and tooling
3. Unstructured policy data
Key insights from user research that will inform future design decisions.
Today, investigators use google to find relevant policy information. They
naturally
A deep dive into users’ current workflow revealed an opportunity.
Users’ needs are not met by Google for the search algorithm focuses on
.
Will a
oriented system better suits our users?
A PoC was developed and tested by a group of users. We collected user feedback and identified a few areas for improvements.
“Testing data has shown that 80% of the documents returned in the first 25 results are relevant. That’s pretty good.”
“I understand users’ frustration comes from seeing completely “non-sense” documents in the top results. I think we need to work together to resolve this issue.”
“Okay, what I can do is implementing a new precision-first model on the top of the current model. By doing that, we can minimize the possibility of showing noice among the first 25 search results.”
“That sounds great! What I can do is creating a way that enable users to remove irrelevant documents. This will not only reduce noice but also give users the ability to curate a list of document that can be directly exported to their case management system.”
“On the other hand, we can implicitly collect feedback data and feed it back to the machine for it to learn. That will be amazing. ”
“Awesome, let’s start with redesigning Document Cards on the search results page. Users need to quickly decide the relevancy of the documents and remove irrelevant ones.”
Pain Point
"I am seeing a lot of
.
Some documents
on the first page are completely irrelevant to my search.”
- User A
“I think we need to ask users to correct the system when the search expansion has gone wrong. The search expansion is machine’s way of understanding and interpretting users’ search terms. ”
“Completely agree. If we can show users how the system works, they will be more inclined to refine the search query and receive better results.”
“The key is to identify the core concept for a search. In most cases, the machine can correctly extract the core phrase from users’ search input. And the entire search expansion is built on top of that. If that premises was wrong, it will dramatically influence the search quality.”
“This is an interesting design challenge. It is an abstract concept that requires a lot of verbal explanations. Should this be only reserved for power users given our user demographic? I think we have a bit of user research to do here. I will create a few concepts and test with our users.”
“That makes sense. I can provide you with a few real-world examples that include edge cases.”
“Thank you! I will make sure that the testing covers some of the edge cases as well.”
Pain Point
"I am afraid of changing
the
.
I don’t know what’s going to happen.”
- User B
