✨ Learn more about our new AI fairness feature!
Case Study:
Exceed.ai, a conversational marketing platform, has been leveraging Mona's AI monitoring solution since 2021. Using Mona, Exceed's data science team has been able to provide confidence in their AI models by immediately detecting and troubleshooting AI issues, preventing any negative impact to their business. As AI solutions are core to their business, ensuring that their models are operating properly is the most important thing for their company. Below, we have detailed the motivations behind the team's adoption of an AI monitoring solution, the process to implement Mona, and the journey they have gone through.
Businesses spend a lot of investment obtaining leads but when SDRs do not have the capacity to nurture potential prospects, they end up missing out on tremendous value. In 2016, Exceed was founded in Tel Aviv, Israel to solve that main business challenge. Exceed provides an AI-assistant that engages with every incoming lead, attends to every inquiry, and qualifies leads at scale through a 2-way conversation by chat, SMS, and email.
Igal Mazor is the Chief Data Scientist at Exceed where he manages the team of data scientists. Using a powerful AI engine, Igal’s team has identified over 25 different intent options that automatically handle the interactions with their client’s customers, accelerating the qualification process to a sales ready lead. Analyzing data from their customers, the data science team had to ensure the models were set up accurately to determine the intent in any prospect’s message, such as setting up a meeting, conveying different levels of interest, wanting to unsubscribe from the database and more, in order to utilize AI to communicate with prospects. Businesses across multiple industries are adopting Exceed’s solution to help their sales and marketing departments save significant amounts of time, build relationships with their customers, and to save money. Exceed is streamlining workflows and automating tasks that SDRs do not have the time to do, resulting in revenue expansion with the increased number of meetings scheduled and reduced number of SDR hours required.
Exceed has customers that span across a variety of industries from fitness/lifestyle to automotive, and more. As more customers use Exceed’s solution for their business, more models are being deployed. Once a new customer joins the platform, the data science team defines targets for the model and applies the generic model with the most generic customer expressions (e.g., “I am interested in” or “I am not interested in”) to determine intent. Then, the team will assess the performance of the model and begin to optimize according to the initial results. If the model is not performing well, the team will start to add industry-specific customer expressions and create a custom model for this certain customer, which includes the same goals as the original model. This iterative process has proven critical to Exceed. However, as the business continues to scale, it has become extremely time consuming for the small data science team, in particular, having to track changes manually between model versions, which has become an exceedingly tedious task.
Recently, the data science team began to notice that there is a lot of uncertainty when it comes to data flowing through the system. There are so many different variables that can change, which immediately impacts the way that the model performs. For instance, details such as the way a person speaks relative to their location, to a new customer from a recently developed business vertical, and all the way to a major pandemic that affects how people communicate. These are all changes that can cause concept drifts and consequently potential issues in the AI system. The data science team has to understand the performance of the model, especially with a new customer, in order to ensure that the responses are accurate or else it will negatively affect the business. With the high number of models that are deployed in production, Igal knew it was necessary to deploy a monitoring solution to ensure that if any issues were to arise, it would be instantly detected and troubleshooted in real time.
Understanding the importance of AI monitoring, Igal began to contemplate an in-house solution. Intuitively, he knew that developing a solution from scratch would be exceptionally time consuming and still will never be as good as a targeted industry solution built by monitoring experts. At just the right time, Igal was introduced to Mona, an intelligent monitoring solution that tracks model performance, provides insights into underperformance issues, and instantly alerts teams of any data anomalies. Knowing that this was the solution he needed, Igal decided to evaluate Mona’s platform.
Igal was surprised to learn that Mona’s configuration was relatively easy to implement, and within just a few days, he already had some initial automatic insights in the dashboard. After a one month trial period, Igal was satisfied with the overall platform and the features that Mona had to offer. He was looking for a fully built, all-in-one monitoring solution that was also affordable for their business. Igal determined that Mona’s solution was mature enough and that it can provide a lot of value to Exceed’s overall business.
Exceed’s data science team is very satisfied with the quality of support that Mona’s team is able to provide them. Working directly with Mona’s Customer Support team, Igal is able to discuss and prioritize new features that will provide additional value for his team. Within just a few weeks, Mona developed two key new features that vastly improved their workflow - smart weekly trends detection and smart dynamic baseline data choosing. The smart dynamic baseline data choosing enables the data team to compare different models within different customer data environments. Both of these features make Mona’s insights more relevant and actionable for Exceed’s data science team, allowing them to respond to changes in data even when those occur only for specific customers or model versions.
In the past year, Exceed has seen massive value using Mona’s AI monitoring solution. The bottom line is - Exceed’s data science team has become more efficient with their time, and they can now catch more potential issues with their data and models, and they do so earlier and proactively.
First, the data science team uses Mona to detect anomalies in their models across the different use cases that they have implemented and also to track if the scores of the model suddenly change compared to the average from a previous time. Mona’s platform provides instant alerts and possible explanations on areas to focus where the issues occur. Even in the most granular data, Mona is able to detect changes within any segment and will be the first to know before any human can detect the issue.
Second, the data science team benefits from the instant alerts that are sent to the individual team members of any sudden changes within the models. For instance, one Exceed customer has multiple use cases and one of the use cases ended up having a score that behaved differently than the others. Immediately notified that the model was underperforming for this customer, the data science team was able to act immediately to optimize the performance of the model. This saved them a significant amount of time by avoiding the team having to manually troubleshoot the issue and also reduced the risk to any of their business KPIs. Mona’s solution ensures that the model deployed fits for the customer, providing confidence in the performance.
Another benefit of using Mona is the flexibility that the platform has to offer since they are able to detect anomalies across all the different use cases, versions, and customers. Mona was able to solve Exceed’s past challenges of having to manually track the specialized models for customers with high variants. Each customer has their own specific model for different use cases that needs to be monitored and Mona’s platform allows for that flexibility.
Mona’s configuration and alerting mechanisms allow the customer success team to utilize the platform in a separate way than the data science team given their different needs. Exceed’s customer success team utilizes Mona to monitor any sudden changes within the data, such as a sudden drop in traffic or if a customer completely stopped using the product. Mona provides them with actionable insights that helps the customer success team retain their customers.
Although there are other advanced features that Igal has not had time to look at, he plans to do a deeper dive into Mona’s platform and take advantage of the other features. Igal notes that there’s more value to Mona’s platform that they are currently not utilizing but they did prioritize the sudden changes within their data as the top priority for a monitoring solution. But for now, Mona’s platform is doing exactly as he intends it to do.
