Chatbots have improved considerably in recent years by leveraging the power of machine learning and neural networks. But even today, most of them work in specific scenarios and not a lot of them can confabulate over a wide range of topics. That might change with Google's Meena.
In the paper "Towards a Human-like Open-Domain Chatbot", Google claims that it has made a chatbot that is superior to all chatbots today. After training on 341GB of social-media conversations and sporting a 2.6 billion parameter end-to-end trained neural conversational model, "Meena can conduct conversations that are more sensible and specific than existing state-of-the-art chatbots.", the researchers wrote.
A traditional metric for gauging the realism of chatbots has been the Turing Test. But research over the years, particularly the Chinese Room Argument, has shown the demerits of the test. Consequently, Meena was tested with Sensibleness and Specificity Average (SSA), a new metric developed by Google that "captures basic, but important attributes for human conversation" in the context of the ensuing conversation.
Impressively, Meena scored well with a 79% on the new metric. This score places Google's new chatbot between the venerable Mitsuko which scored 56% and a human conversing partner who score 86% on the same test.
But Google will not be releasing Meena's demo to the general public domain before putting its safety and potential biases to the test. If all checks are a go, the company might release a demo in the coming months to help advance research into neural conversation models and chatbots as a whole. The firm also noted that it "focused solely on sensibleness and specificity in this work" but other key attributes like personality and factuality might be the target of subsequent projects.
If you are interested in finding out more, you can study the paper published at arXiv.
5 Comments - Add comment