Conversational AI Evaluation Guide

A Personalized Conversational UI For Your Apis
Get Started


Conversational AI Evaluation Guide
As with every project, you are setting out to solve a problem but more importantly you want to make sure that it can be solved in the most efficient way possible, creating a competitive advantage.

Many companies have trouble getting their digital assistant projects to production or if they make it to production, they struggle with adoption and user engagement. However, the largest group consists of organizations that struggle to simply get management's buy-in to invest in the project. Whichever bucket you fall into, this guide will provide you a concrete understanding of how to tackle these challenges and safeguard your investment.

This evaluation guide is intended to help you learn about all the facts of Conversational AI so that you are well equipped to create a plan to design, build, and grow your technology investments.

Stages Of Developing A Conversational AI Strategy

Conversational AI Evaluation Guide
You are trying to build a business case to launch a new digital assistant or chatbot.
Conversational AI Evaluation Guide
You have received buy-in to move forward on a new project and you are at the place where you need to select vendors and start building a team.
Conversational AI Evaluation Guide
You have launched a new assistant, but are struggling to get the type of engagement or transparent ROI.
Conversational AI Evaluation Guide
You have selected your vendors, but need resources and a roadmap to maintain your project.

Developing a multi-channel, multi-language, 24-hour Brand Ambassador that scales with low latency, high containment and enough personality to create interest but ultimately an experience that works well is not easy. That's why we're here to help.

The Business Value of Conversational AI

Conversational AI Evaluation Guide

Users value Digital Assistants because they are fast, intuitive and convenient. For enterprises, AI chatbots offer a way to build a more personalized and engaging customer experience, which in return delivers a wealth of customer information that is highly valuable.

Ultimately giving you a better understanding of your customers and allowing you to seamlessly grow your business.

Take a look at 7 ways to derive value from Conversational AI:

The Biggest Value Is the Data
Interactions between a chatbot and a customer will throw up new insights into customer preferences and even offer a window into their thought-process and decision-making.

Immediate Response

Speed and convenience win over customers today, far more than the price. 75% of customers expect "now" service within five minutes of making contact online.

Reach New Channels

This technology scales quicker than a back-office team. If you want to reach customers through newer channels, then you don't have to worry about how you're going to pull together another team or calculate how much load your current team has. Just plug and play and add channels without scaling up teams.

Increase Engagement

Customers want service now and they expect it to be available 24/7/365. They want to message you a question while waiting in line for a coffee or use voice to make an online purchase while driving to work - and they want to do so using all of the devices and services they already use every day.

Drive More Revenue

Conversational AI is built on the same technology as recommendation engines - like the ones on Amazon that just so happen to know things you might be interested in.
A conversation allows your brand to drive the next best action, better than a website can.

Reduce Costs

Service more customers without increasing your overhead. Virtual customer assistants can help curtail inbound queries by up to 40%, and often deliver first call resolution (FCR) rates far in excess of live agents.

Maximize Staff Skills

By automating a proportion of the calls, emails, SMS and messages that would have otherwise required direct human involvement, a digital assistant can free up time to allow existing employees to focus on higher-value customer engagements.

Architecture Overview

Conversational AI Evaluation Guide

A user interacts through a channel by sending a message to a system in order to get something done or to access knowledge.

Once the channel has received the message whether it's in voice or text format, it is processed into text along with metadata about the channel and the user (if available) and sent to the Natural Language Processing (NLP) system.

The purpose of the NLP system is to determine what the user is requesting by classifying it into a pre-built list of intents and then sends the results of its analysis to the business logic component to logically determine how to fulfill the request with the resources available.

The business logic server uses the intents and slots from the NLP system as a sort of command to go fetch the data necessary for the response. If enough data is not available, then submit a request back to the user to do one of the following:
Clarify their query by providing an additional piece of data,
Disambiguate conflicting data,
Steer the user to a similar command, or
Instruct the user to retry altogether.
Once the business logic server has determined its next best action, it will communicate that back to the NLP system to formulate a proper response for the channel, submitting that response back to the user.

Features Of A Great Conversational AI System

Truly Conversational
Enterprise-Grade Solution
Unique Personalization
Data Ownership and Analytics
Developmental Control
Hybrid Model
Data Security
Cross Platform
Brand Differentiation

It may seem obvious but there's a world of difference between a chatbot answering a question and holding an intelligent conversation. An engaging exchange will not only improve the customer experience but deliver the data to help increase your bottom line. To achieve this, the user interface needs to be as humanlike and conversational as possible.

A conversational chatbot must understand the user's goals, no matter how complex the sentence, and be able to ask questions to remove ambiguity or discover more about the user. It needs memory to reuse key pieces of information throughout the conversation for context or personalization purposes and be able to bring the conversation back on track, when the user asks off topic questions.

If you are a multi-national company, it becomes imperative for you to have a chatbot development platform of your choice to do all this, and in your customer's native language too.

Few chatbot development platforms were built with the enterprise in mind. Consequently, features you might expect as standard such as version control, roll-back capabilities or user roles to manage collaboration over disparate teams are missing.

In addition, look for features that will aid the speed of development including automated coding, web-hooks to allow flexible integration with external systems, and ease of portability to new services, devices and languages.

Personalizing an automated conversation, whether it's simply accessing account information to answer a billing query or taking into consideration the customer's love for Italian food when recommending a restaurant, not only delivers a more accurate response, it increases engagement too.

While some information can be learned 'explicitly' (such as the customer choosing a preference from a list of features), it's the automated learning through 'implicit' methods (like information gleaned from, previous interactions) that really harnesses the power of Conversational AI. This can then be combined with other information and data sources such as geo-location, purchase history, and even time of day, to personalize the conversation even further.

One of the key considerations in choosing a chatbot platform is data. People reveal vast amounts of information in everyday conversations.

Their individual preferences, views, opinions, feelings, inclinations and more are all part of the conversation. This information can then be used to:

Make the conversation relevant and more engaging
Train and maintain your conversational AI chatbot interface
Analyze data to deliver actionable business insights

That's why it's so important that enterprises maintain ownership of their data. You will be surprised to find several chatbot development tools that allow businesses to create chatbots but don't actually provide any of the details of the conversation, but simply delivers the outcome -such as that final pizza delivery order. Alongside data ownership, carefully consider the data analytics package provided as part of the platform, including the flexibility in drilling down the information and understanding the context of conversations, as well as the level of details provided.

It's very difficult to anticipate how people might use, or abuse, an AI application. Certainly, Microsoft didn't envisage that "helpful" members of the public would teach Tay to start Tweeting inappropriate messages. FEATURES

Microsoft created Tay, an AI Bot with the personality of a teenage girl to "experiment with and conduct research on conversational understanding." Tay was designed as a showcase of machine learning and Natural Language Processing (NLP) but unfortunately very neatly illustrated the problem with some conversational AI development tools that lack the control required to supervise the behavior.

By ensuring a level of control within the application, enterprises can not only avoid awkward mistakes, but provide a 'safety net' for managing unexpected exceptions during a conversation, always ensuring a smooth customer experience.

Most chatbot platform development tools today are either purely linguistic or machine learning models. Both have their drawbacks. Machine learning systems function, as far as the developer is concerned, as a black box that cannot work without massive amounts of perfectly curated training data; something few enterprises have.

While linguistic-based conversational systems, which require humans to craft the rules and responses, cannot respond to what it doesn't know, using statistical data in the same way as a machine learning system can. A hybrid approach that contains machine learning and linguistic models is best and allows enterprises to quickly build AI applications regardless of their starting point - with or without data - and then use real-life inputs to optimize the application from day one. In addition, it ensures that the system maintains a consistent and correct personality and behavior aligned with business goals.

Data security is a key consideration for any enterprise, particularly when dealing with regulatory frameworks and customers' personal information. Flexibility is essential in an AI chatbot platform to meet today's exacting security conditions, across multiple geographies and legal requirements.

Widespread cloud adoption by corporates across industries is a big positive but regulations in certain industries and stringent security policies might make it worth your while to ensure that an on premise option is available as well.

Conversational applications are gradually infiltrating all aspects of everyday life, so it makes sense to ensure that conversational applications can be easily ported to existing and future devices. 14

While it's easy to state that applications can be built to run on a variety of platforms or services, all too frequently each one requires a completely newbuild. Investigating how much of the original build can be reused at the start may save significant resources in the long term.

It's also worth looking at how the application will support your users as they swap from device to device during the day. Seamless persistence of conversations increases engagement and customer satisfaction.

By adding an intelligent conversational UI into mobile apps, smartwatches, speakers and more, organizations can truly differentiate themselves from their competitors while increasing efficiency. Customization offers a way to extend a brand identity and personality from the purely visual into real actions.

Building a Business Case

In some cases, it's easy to start with a proof of concept (POC) and then use the POC to solidify the business case.

Some use cases won't require much heavy lifting or a sophisticated suite of products. Take for instance a FAQ chatbot on a website with less than a million monthly unique users. A project like this will fall into the small to mid-size scope and falls well under 100k per year in licensing and resource costs. It's important to keep in mind that some projects can also go well over $3 million per year. Having an idea of your business case will make this evaluation guide much more useful for you.

Contact Center - Save money and do more
Customer Experience - Provide better tools for your customers
At What Size Should You Be Thinking About Automating a Process?

The size of the organization does NOT drive whether the organization can benefit from intelligent automation.

Rather, it's the amount of time dedicated to a given business process that matters in determining the effectiveness of automation.

As a rule of thumb, once a process is being limited by hours of availability or gets buckled during spikes of activity, it is time to seriously consider applying intelligent automation to perform the task.

Conversational AI Evaluation Guide

Key Questions to Ask

Some Key Questions to Ask for Problem Solving
What bottlenecks are negatively impacting operations?
What are the complex business processes that lead to the bottlenecks?
Where are our employees spending most of their time?
What regular tasks are repetitive?
Where are our employees interfacing with a variety of systems to complete a process?
Some Key Questions to Ask for Optimization
What KPIs matter most to us?
What are the complex business processes that lead to the bottlenecks?
Where are our employees spending most of their time?
Where is staff performing repetitive and administrative tasks?

Selecting Your Channel(s)

Conversational AI Evaluation Guide

There are different design approaches that can be used to determine the best channel for your assistant. You can start with building out personas or you might be lucky and happen to already understand what channel has the best reach for your user base. Here we've outlined an indicative list of criteria for selecting a channel:

If a bot is going to be deployed within an organization for communication between employees, then select a channel that is used by the majority of stakeholders in your business, like Microsoft Teams.
If your user personas predominantly use voice over text chats, then selecting Alexa or Google Assistant will make more sense than focusing on other text-based channels.
Channel selection depends on your business model as well. For instance: if you are a consumer business (B2C), then your priority should be having presence in channels such as SMS, WhatsApp, Facebook Messenger over having a chatbot integrated with your website. However, it is crucial for a B2B business to have a robust website with an integrated chatbot. WhatsApp and Facebook messenger might not be pertinent for a B2B business model. Also, you will want to consider what type of user interactions are best for each channel and how you are going to do a phased roll out of the channel. A phased roll out of your channels is best in almost every case because it gives you time to focus on optimizing each channel and to enhance the user experience before it's put to the test.

Natural Language Processing (NLP)

Our Application Solutions, Your Business Targets

Conversational AI Evaluation Guide

There is a plethora of vendors that provide NLP platforms, some of these vendors have very specific niche offerings for a particular market. The reason why you see these vendors focus on a market is because the customizations can be steep if you are starting from scratch and they can offer an out of the box solution. A simple example is Banking. A vendor with focus on this market can offer a predefined set of intents that will help in the design phase of building. Regardless of the vendor's market focus, there are a few features to look for and technical approaches:

Deep learning systems are inference based and make their best guess on the correct way to "classify" messages in a conversation.

This classification is done by vectoring - imagine a large 3D space that can map a query into coordinates and based on intents that live close to that query is how the system does its classification. There are drawbacks to this approach - one of the biggest hurdles is that deep learning is a black box, because it's inference based it's impossible to forensically analyze how it came to the decisions that it did, so the only way to fix a model is to curate the data that you use to train it and then test the result… repeat until the system has been fined tuned. This is why data curation is so important to training deep learning systems.

These systems are focused on traditional linguistic approaches that include parts of speech tagging and tokenization. This approach relies heavily on keyword detection and manually curating all the possible types of queries a user may have.

What we are seeing now is a mix between traditional ontological approaches and deep learning. This approach enables deep learning components to understand the meaning of entities and their relationship to the rules of the physical world. It also provides an intelligent way to personalize a virtual assistant therefore maximizing the end users experience.

Poc's, Pilots and Production

Conversational AI Evaluation Guide

Beginning with a POC and moving to a pilot and finally production is a common path to take when tackling a new virtual assistant and project. Proof of Concepts are extremely valuable to help pitch the project internally and to also validate the use case on the smallest scale. Moving from the Proof of Concept to the pilot is a huge step because the pilot really should be considered a starting point for production. By having a great start and understanding what you are trying to get out of the pilot will help tee-up the production stage for even more success. Production, the goal here is to initially capture the features that add the most value and keep the system free from too much complexity to create a sense of usefulness and intrigue.

The tricky part is that when new skills are added through the iteration while in production, it can oftentimes be difficult to regain users trust when things don't go as planned. It's key to right-size the initial production project, as is the case in any project even outside of Conversational AI. However, assistants are subject to the same evaluation criteria as humans when they first start a new job but even a bit more difficult because we don't give these new systems the benefit of the doubt.


What is the underlying desired result of a query?
What is the user trying to do or what knowledge is the user trying to access?
Simply put, goals of the user in pursuing a conversation with a digital assistant.
Natural Language Processing - this is the process of identifying what the user query or message is and breaking down the message into components for better understanding the user's intents.
Automatic Speech Recognition
Text to Speech
Dialog Repair
When the system misinterprets a user and the user needs a method to correct the system on the mistake
Conversational Healing
Another name for Dialog Repair
Co-reference resolution is the task of finding all expressions that refer to the same entity in a text.
Context Awareness
A system that can keep track of the entire conversation and use pieces from an earlier query or message in the current query
Deep Learning
Adjusts the style of the current speaker with a speaking style
Data Collection
A process for training machine learning algorithms in NLP, TTS and ASR systems
Response Logic
The logical set of instructions for the content that is displayed back to the user. This content can be personalized and contextual - these features can be used to drive a better user experience.
Business Logic
How to process intents, where to pull data, how to format the data so that it can correctly talk to all the systems in the conversational AI pipeline.
Similar to a noun in language terms. A person, a place, or a thing
Another name for Entity. Most commonly used by Amazon Lex and Google's Dialog Flow
A technical name for how to flow context throughout a conversation between two or more intents
Conformational Competency
An intent that requires certain pieces of information (slots) to be filled in order to fulfill a goal. Example: " I would like to open up a new bank account "
Informational Competency
A simple intent that does not require the user to fill in the missing information. An example would be "hello, what is your name".
Information that does not contain any system or structure for categorizing or organizing information in a systematic way for a computer to understand.
Information that has limited markup or metadata that can be used to guide a system
Amazon Mechanical Turk
A service by amazon for collecting data from a crowd of service workers
A tool for deploying code in a containerized fashion

Conversational AI in Production (End User Apps)

Company Name
Capital One
Eno monitors your account and automatically notifies you about unusual charges, free trials and more
Learn more
AI Dungeon
AI Dungeon is an AI-powered text adventure where every response is determined by an AI language model. The first of its kind in which any story option is possible, and the AI adapts the adventure to the users’ input. The game sees 20-25,000 daily users.
Learn more
Replika was founded by Eugenia Kuyda with the idea to create a personal AI that would help you express and witness yourself by offering a helpful conversation.
Learn more
By combining OpenAI's technology with the in-depth work and research Quizlet is doing with machine learning, Quizlet will be able to develop example sentences for people studying vocabulary and languages, the way a tutor does, to help students integrate their knowledge in a fun way and test themselves more comprehensively. Using the latest in NLP technologies allows Quizlet to build toward the future of an AI-powered tutor in your pocket.
Learn more
Koko is using OpenAI's technology to enhance its AI capabilities and improve its ability to keep users safe. Using the API, Koko can automatically identify users in acute states of crisis and route them to specialized services (such as the National Suicide Prevention Lifeline).
Learn more

AI Automation Workshop

Softura can help your organization create a "Digital Workforce." We use software designed to model and emulate human job roles by performing end-to-end job activities using automation and AI-based skills and Natural Language Processing. All companies are looking into their roadmap to deploy AI Automations to lower cost and increase efficiency. AI Automations also contribute to growth by freeing human employees time so they can perform higher value work.

This FREE workshop is a great way to bring key stakeholders of your business to a common forum where we identify processes that can benefit from automation and prioritize use cases where a digital workforce can be leveraged to drive efficiencies. To learn more about the workshop such as the agenda and key takeaways, visit our workshop page:

AI Automation Workshop

About Softura

Softura is a global software development and modern consulting company that leverages more than 20 years of success to design, code and deliver complex architecture, applications and systems built on leading modern technologies. Our proven record of consistently providing cutting edge technology solutions to solve complex business problems and drive key outcomes has led us to become a trusted technology solutions partner for hundreds of mid-market and enterprise clients.

Conversational AI Evaluation Guide
Conversational AI Evaluation Guide
Conversational AI Evaluation Guide
Conversational AI Evaluation Guide
© 2024 Softura - All Rights Reserved
cross linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram