Converting the Point of View of Messages Spoken to Virtual Assistants

TL;DR

This paper presents a system that converts the point of view in messages spoken to virtual assistants, improving message understanding and delivery through rule-based and neural approaches, and evaluates their effectiveness with multiple metrics.

Contribution

It introduces a novel system combining rule-based and neural models for point-of-view conversion in virtual assistant messages, along with a new dataset and comprehensive evaluation metrics.

Findings

T5 achieved the highest faithfulness scores with BLEU 63.8 and METEOR 83.0.

CopyNet was the most natural model with a relative perplexity of 1.59.

The publicly released dataset contains 46,565 crowd-sourced samples.

Abstract

Virtual Assistants can be quite literal at times. If the user says "tell Bob I love him," most virtual assistants will extract the message "I love him" and send it to the user's contact named Bob, rather than properly converting the message to "I love you." We designed a system to allow virtual assistants to take a voice message from one user, convert the point of view of the message, and then deliver the result to its target user. We developed a rule-based model, which integrates a linear text classification model, part-of-speech tagging, and constituency parsing with rule-based transformation methods. We also investigated Neural Machine Translation (NMT) approaches, including LSTMs, CopyNet, and T5. We explored 5 metrics to gauge both naturalness and faithfulness automatically, and we chose to use BLEU plus METEOR for faithfulness and relative perplexity using a separately trained language model (GPT) for naturalness. Transformer-Copynet and T5 performed similarly on faithfulness metrics, with T5 achieving slight edge, a BLEU score of 63.8 and a METEOR score of 83.0. CopyNet was the most natural, with a relative perplexity of 1.59. CopyNet also has 37 times fewer parameters than T5. We have publicly released our dataset, which is composed of 46,565 crowd-sourced samples.