Your chatbot’s replies will vary widely depending on which AI model you choose. In this guide, we’ll share tips for choosing the best one for your website.
How to evaluate AI models
Before you begin, make sure that you have finished training your chatbot so that it is prepared to answer your sample questions.
First, navigate to the Resolver > Models menu. Then, open your chatbot at the bottom-right of the screen and ask it a question.
Then, use the Chat Model option to choose a new model and save your changes. Open the chatbot again, but this time, click the button in the widget to start a new chat.
This ensures that your chatbot starts from a clean slate. Otherwise, its reply will be influenced by the previous messages. Now, you can copy and paste the same question from before into the widget and evaluate the chatbot’s reply using this new model.
Repeat these steps for each model that you are interested in trying.
Once you’re finished, navigate to the Chats menu, where you can easily flip between chats to compare your chatbot’s replies.
You can repeat this process as many times as is necessary to select the best model.
Asking better questions
When testing your chatbot, you may be tempted to feed it softball questions, like “how long does shipping take?”
While that’s an okay way to get started, to really test the chatbot, it’s best to copy real questions from your customers. Find recent chats or emails you’ve had with your customers, and copy their initial message verbatim – misspellings and all. This is the best way to see how the chatbot will fare in real conditions.
Balancing cost vs. quality
The price from one AI model to the next varies greatly. The most expensive model may be anywhere from 4-10x the cost of the cheapest model.
For a customer support chatbot, the job is pretty simple. It’s going to read the customer’s query, read the most relevant text from the training data, and then create a reply. It doesn’t need deep reasoning skills.
We include more advanced models in case people want to use them, but for most sites, they’re not necessary. The more advanced models also take longer to generate a reply.
If you are seeing slightly better results from the most expensive model vs. the cheapest, it’s likely best to go with the cheaper model. With that said, if your site only gets 1-2 chats per day, the price doesn’t really matter; you won’t spend more than a few dollars in any given month even with a cutting-edge reasoning model.