This website uses cookies to store information on your computer. Some of these cookies are used for visitor analysis, others are essential to making our site function properly and improve the user experience. By using this site, you consent to the placement of these cookies. Click Accept to consent and dismiss this message or Deny to leave this website. Read our Privacy Statement for more.
Test | Print Page | Contact Us | Your Cart | Sign In
Patient Experience Case Study - UCHealth
Share |

See the opportunities and challenges organizations face in addressing the patient experience.

Case Studies provide real stories of current efforts, including programs being initiated, practices being implemented, and outcomes being targeted and/or achieved. Case studies are presented as both an opportunity for learning from others as well as a spark for further ideas on how we work to improve the patient experience. 

Have a case study to share? Download Case Study Template.

Interested in receiving PX WeeklySubscribe here.

Conversational AI and COVID-19: How our virtual assistant helped us meet the needs of our patients when they needed us the most

Key Takeaways 

  1. Since the start of the COVID-19 pandemic, there has been a significant shift to virtual and digital technology for healthcare delivery, and conversational AI can support the change.

  2. COVID-19 has put a spotlight on digital health tools such as virtual visits, online scheduling, electronic check-in and 24/7 access to personal health information through virtual assistants. In the first five months of the pandemic, patients asked our virtual assistant more than 11,000 questions mentioning the virus, and about 7,000 questions around virtual visits.

  3. Consumers are getting increasingly technologically savvy and, in the case of COVID-19, better informed before they arrive to a health system’s resources. Conversational AI allows us to meet patients’ needs while offering answers and listening to their concerns and making changes on the fly.

Experience Framework Alignment

  • Innovation and Technology


What was the opportunity, issue or challenge you were trying to address and in what setting?

Digital health is at its best when it uses technology as a catalyst for greater access without diminishing human connection.

The COVID-19 pandemic has put a significant spotlight on digital health tools such as Virtual Visits, online scheduling, electronic check-in and 24/7 access to personal health information through virtual assistants. That increased visibility and use, coupled with increased anxiety and fear of the virus, makes connection even more important.  

Since this was a global issue, we asked ourselves: How do we leverage our existing technology to listen to patients’ needs during the pandemic and deliver a response tailored to those needs?

The answer was our virtual assistant “Livi,” a conversational AI tool that has been improving digital communication and patient experience at UCHealth since 2017.


What process did you use to develop a solution?

UCHealth’s strategy for updates to Livi relies on consistent monitoring of user queries, taking special note of queries that Livi doesn’t comprehend or doesn’t comprehend fully. These updates are usually published weekly – both the responses Livi offers to our users and the training data that leads users to those responses. 

We have an internal dyad focused on the day-to-day monitoring. They work together to monitor user behavior and questions and make changes accordingly. Although the Experience Innovation team owns the overall strategy, changes and updates are always a team effort between our Livi ownership dyad, our IT department Epic analysts and application developers, and our vendor, Avaamo.

Livi’s use during the early weeks of the COVID-19 pandemic grew significantly – at its peak in the last week of March 2020, there were 4,890 users, or nearly 250% more than the number seen during the first week of January. The dramatic increase yielded a tremendous amount of data on what users were looking for and how Livi fit into a media and informational environment that, at the time, was heavily focused on COVID-19 infection rates and symptoms.


What outcomes were you looking to achieve? 

Our primary concern was, and continues to be, to provide useful information to as many people with as little confusion as possible.

In this case, due to the importance of COVID-19 information to the health and safety to our users, it meant respecting the near constant changes to that information. Guidance from global, national and statewide health authorities became outdated within the span of a day, and we did not want to risk disseminating anything that was inaccurate.

Our system strategy to use a single source of truth for general purposes allowed us to continually provide up-to-date information – and allowed us the bandwidth to focus more intently on providing resources for the majority of users whose queries were more specific.
Using our existing maintenance process, we made changes and tweaks to Livi’s responses based on what users asked of her – reacting quickly, in near real-time, to the needs we were seeing while we monitored those conversations every day.


What specific steps did you take to address the problem? 

The first query that mentioned COVID-19 or coronavirus happened on Feb. 19, 2020, when a user asked: "Should I be worried about coronavirus?"

Livi's response to this query was a retrieval of information from a health library connected to This is the default behavior when Livi doesn't understand a user query but recognizes a possible search term. At the time of this query, the library had limited information on coronaviruses generally, but nothing specific to the coronavirus responsible for COVID-19.

The first step to a more effective conversational interface around any topic is to account for a variety of terms and idiosyncratic spellings. For COVID-19, we created a dictionary entry with a wide array of variants, – “coronavirus,” “corona virus,” “corvid19,” etc. – based on queries from users with varying levels of familiarity with term, especially in the earliest weeks of the pandemic.

When it came to crafting the substance of a response, we again examined queries for insight into what would be helpful. Users did not, as a rule, come to Livi for general information on COVID-19 or symptoms. Though these queries did happen (e.g., “What information do you have on Corona virus” on Feb. 22), they happened most often only in the earliest days of the pandemic. Much more common were queries about the specific users’ health or UCHealth as an institution. For instance:

"Find a covid-19 testing site."
"Wanna get tested for Corona virus."
"I went to get tested for covid 19 on Friday April 17th and was told I could get results within 24 hours. “I still haven’t received notice. Can you help?"
"Hi Livi, I would like to do a virtual visit. I’m experiencing some symptoms of Covid-19 and have traveled recently."

The weekly number of queries mentioning “coronavirus” or “COVID” significantly increased after March 11, peaking during the week of April 1. These queries continued to be very specific to the individual user’s health and UCHealth. In many cases, users appeared educated about symptoms before coming to a UCHealth resource (e.g., “Corona virus testing for 74-year-old individual with symptoms including low-grade fever. Are they supposed to call their doctor and be tested?” on March 4).

Keeping in mind our goal of providing useful information to as many people with as little confusion as possible, as well as the gravity and fast-pace of the COVID-19 outbreak’s first weeks, we needed to also acknowledge the limitations of the tool. The natural language processing behind Livi improves regularly, making significant apparent gains even in the few months preceding the pandemic. But it is still an imperfect technology. The risk of users getting unhelpful responses was not negligible.

For these reasons, we decided to move forward with a single resource card – a wide net to catch as many users as possible with links and calls to action that reflected what would be most helpful.

Those queries (for instance, “Do you have a coronavirus testing site?”) and some variants were added to the Livi training data for this resource, to ensure that users with these types of queries would always match to this response. The response resource for COVID-19 included:

A link to the UCHealth COVID-19 information hub – the central source of truth for UCHealth COVID-19 news, news, policies, a CDC symptom checker, and other resources.

o If users were on the mobile app or using the desktop patient portal (My Health Connection), linking to the information hub on the main site also allowed those users to see the latest updates on visitor policies, virtual visits, etc.

A link to the UCHealth Today main page for the latest articles with useful information (tips for disinfecting your home during the pandemic, recipes, at-home remedies for symptoms, etc.).

An action button of Schedule Virtual Urgent Care, which triggers Livi's "schedule an appointment" conversation flow.

An action button of Send a Message to My Doctor, which triggers Livi's messaging flow, presenting users with a list of possible message recipients (UCHealth providers who have seen the patient in the past and have enabled messaging).

As the pandemic and its associated shutdowns drew on, user queries changed as well. For instance, the time it took to process COVID-19 tests in March and April 2020 was considerably longer than it is at the time of this writing, July 2020. Testing supplies were also fairly limited. This left a comparatively small number of Livi users to ask about the status of their COVID-19 tests; once test results were quicker to appear in the health record, these queries reduced in number and leveled off. 

Our response to these users’ behavior was to add to Livi’s existing test results retrieval, which allows Livi to display users’ test results in the chat window. Using the COVID-19 dictionary term, we built a dynamic response for users asking specifically about the status of their COVID-19 tests:

“There are delays in testing samples for COVID-19 at both state and commercial testing labs. Some tests might take about a week before we hear results. UCHealth is working to speed up testing. Once your test has been entered in My Health Connection, it will be called COVID-19 (SARS-CoV-2), and your provider will reach out to you.”

Finally, the number of Virtual Visits increased 500-fold, from an average of 20 every day prior to March 1, to an average of 4,000 per day during the month of April. The newness of this care delivery method saw a shift in user behavior as well – whereas most queries mentioning Virtual Visits prior to the pandemic were about scheduling (e.g., “I would like to schedule a virtual visit with my rheumatologist…”), pandemic-era queries shifted toward technical troubleshooting or general information about receiving virtual care. For instance:

“How do virtual visits work?”

“What is the process to get set up on virtual visits”

“I was advised to do a virtual urgent care visit, but I can’t figure out how to make it.”

“Hi Livi, I talked with a nurse and set up a virtual Urgent care visit for 11:20 this morning. I am on my UCHealth connection page. Is there a message or something to push to tell me how that visit is set up?”


As with the COVID-19 resource, we intended this response for as many users with as little confusion as possible, while acknowledging the sheer variety of queries and the limitations of natural language processing. Therefore, we built another wide-net response for Virtual Visits, including:

A link to the virtual visits service main page with information about the program, instructions, and frequently asked questions.

Embedded videos on how to schedule primary care Virtual Visits, responsive to the user's device (computer or mobile), so users on mobile see the mobile tutorial video, and users on desktop see the desktop version.

Embedded videos on how to start the Virtual Visit eCheck-in process, responsive to the user's device (computer or mobile) so users on mobile see the mobile tutorial video, and users on desktop see the desktop version.

An action button of Schedule a Virtual Visit appointment, which begins the schedule an appointment conversation flow. From this conversation, we also included a message to set expectations about operational limitations for online scheduling:

o “Some providers are set up to schedule Virtual Visits directly online. If you're logged in and trying to schedule a Virtual Visit, but the provider you want to see isn't listed, I'd suggest calling their office.”


What resources, if any, did you engage - either internally or externally - to address the problem?

The primary resources used for this process were the Avaamo platform for conversational AI, information pages, and resources in the form of effort from an Avaamo developer and the UCHealth Livi team.


What measures did you establish to determine the success of this effort?

Measures of success for conversational AI are notoriously hard to gauge, but UCHealth uses two primary measures, which are captured weekly:

  • Number of users (both authenticated users signed in to their Epic MyChart account on web or mobile, and guest users who are not signed in).
  • User feedback in the form of thumbs-up/thumbs-down responses after various conversation flows.


What was the ultimate outcome of your effort?

Using the measures above, we saw dramatic favorable increases during and after the initial weeks of the pandemic.

The number of users in the last week of February was 2,841 – in the first week of July, it was 5,339.

Livi’s lifetime positive user feedback was at 59 percent in the last week of February, and in the first week of July it had grown to 75 percent.

We feel that our COVID-19 responses, as a significant piece of our conversational AI efforts during the first months of the pandemic, contributed to overall user growth and satisfaction.


What lessons did you learn would that would share with others as they consider addressing a similar issue?

Always listen to patients’ needs over the bells and whistles of a new gadget promising to improve experience. Always be prepared to fail fast, get back up and make the solution better. The only way to respond to crisis is to listen, move fast, and keep listening.

With each passing year, more industries are shifting to some form of conversational AI to improve customer experience. We have learned that you need to be patient, have a hyper focus on what the customer wants from the technology (not what we think they want), and build the pieces one by one. That approach was a significant help when we needed to respond to the COVID-19 pandemic. 

Whether it is responding to a pandemic or day-to-day activity from patients, using conversational AI is ultimately about offering convenient access to information, so patients can focus on the relationships that matter.

About UCHealth:

UCHealth is a nationally recognized, nonprofit network of 12 acute care hospitals and more than 150 clinics throughout Colorado, southern Wyoming and western Nebraska. With University of Colorado Hospital as its academic anchor, UCHealth is uniquely able to provide advanced treatments and innovative clinical trials, ensuring excellent care and outcomes for more than 1.6 million unique patients each year. UCHealth’s mission is to improve the lives of people in Colorado and beyond, and in fiscal year 2018, UCHealth spent $854 million on financial assistance, subsidized care and other areas to directly benefit patients and the communities it serves.

Appendix A.

Appendix B.

For more information contact:

Matt Andazola, Content Manager, Experience and Innovation, UCHealth
Nicole Mossing Caputo, Sr. Director, Experience and Innovation, UCHealth

Denver, CO, USA

Stay Connected

Sign up for our informative series of monthly e-newsletters from The Beryl Institute.

The Beryl Institute
1831 12th Avenue South, #212
Nashville, TN 37203