A Framework for Conversational System Modeling

Introduction There is an interesting but criticized theory in linguistics: Grice's Maxims of Conversation. This theory, introduced by British philosopher Paul Grice, attempts to explain how humans converse. That is, it provides a framework for understanding how effective communication between two people takes place. The basic idea is rather simple: the theory posits that people want to be cooperative when speaking with others. And because they want to be cooperative, they aim to be informative, truthful, relevant, and clear when engaging in human conversation. Putting aside the provability of the theory for the moment, the goal of having an effective conversation is worth exploring in the context of modern AI systems capable of voice interaction. How do these systems, which include popular consumer variants such as Alexa and Siri, "know" how to converse with human users? We can likely agree that they do not have the objective to be cooperative, at least not in a human sense. So, how do these systems know: When to speak? When to remain silent? When to stop speaking due to being interrupted? When to interrupt and begin speaking? You've likely already identified one possible answer: voice interaction models can predict conversational events that allow them to effectively simulate human-to-human conversations. In this case, the task then becomes how the prediction problem for conversational speech is modeled and implemented for speech dialogue systems. This exact challenge was the focus of the recent FinVolution Teach AI When to Speak competition ("the competition"). FinVolution is a Chinese fintech firm founded in 2007 with technologies in credit risk assessment, fraud detection, big data, and artificial intelligence. I participated in the competition and was very impressed by the completeness of the competition setup - the organizers went so far as to provide base training and inference logic. The competition, unsurprisingly, involved Chinese-language conversations. However, the general modeling problem can be adapted to other languages.My goal with this article is to provide you with an understanding of the problem setup, which you can use as a basis for your own experiments in conversational modeling. The information I provide here is essentially my "wrapper" over the publicly available competition description.The competition itself closed in July, and the competition assets are proprietary to the organizers. Although I can't provide the training data and baseline training/inference scripts, I aim to give you enough of a conceptual understanding so that you can take the next step of creating your own training dataset(s) and building your own model(s).Dialogue Modeling To begin developing a model that can predict events in a conversation, we need to first understand what those events could possibly be. Imagine your own conversations, both in-person and not in person (e.g. over a cellphone). In the case of the former, you might listen and look for clues from the other participant in the conversation. For example: Your conversation partner might stop talking, indicating it is your turn to speak. Your conversation partner might use filler words, such as "uh huh", "wow", and "oh" (in English), indicating his/her understanding and/or reaction to what you are saying while simultaneously signaling that you can continue speaking. Your conversation partner might make a physical gesture, such as raising his/her hand up while you are talking, indicating he/she would like to interrupt you so he/she can say something and take their turn with the conversation. Prosodic characteristics, such as a speaker's rhythm, pitch, and loudness, can provide conversational cues. We could go on and on, even going so far as to extend our set to specific cultural behaviors. The point is that we can identify certain sensory events (audio, visual, etc.) that explicitly and implicitly guide us through a conversation. The wide variety of conversational cues drives a corresponding variety in dialogue system design.Figure 1 - Turn-Taking in Conversational Systems (Patamia et al., "Turn-Taking Modeling in Conversational Systems: A Review of Recent Advances", MDPI, 2025) #Figure 1 - Turn-Taking in Conversational Systems (Patamia et al., "Turn-Taking Modeling in Conversational Systems: A Review of Recent Advances", MDPI, 2025) For our purposes here, we will assume that our predictive speech dialogue system only processes audio events - i.e. other sensory cues are not available to be used in prediction. This "limitation" is not fatal given the billions of telephone conversations that occur every year and it allows us to simplify the modeling problem. Moreover, this simplification aligns with FinVolution's conversational model used with the competition:Figure 2 - Two-Participant Conversation Model (FinVolution, "Teach AI When to Speak Competition", 2026) Figure 2 - Two-Participant Conversation Model (FinVolution, "Teach AI When to Speak Competition", 2026) The figure depicts the audio waveforms of two participants engaged in a conversation for a short time window. Five distinct conversational events are defined and captured in the time window:Table 1 - Conversational Events Event Label Event Type Description Example C Continuation A conversation participant, the "current speaker", is speaking and continues speaking. The current speaker is articulating an idea. T Turn Change The other conversation participant has now become the current speaker and has started speaking. The first speaker asks a question, e.g. "How are you today?", and the other person begins speaking to provide an answer. BC Backchannel The conversation participant who is not the current speaker makes a short utterance without a turn change - i.e. without becoming the current speaker. The conversation participant uses a filler word such "uh huh", "wow", or "oh". I Interruption Both conversation participants are speaking at the same time. The current speaker is interrupted by the other conversation participant who starts speaking over him/her, possibly attempting to invoke a turn change. NA Silence Neither conversation participant is speaking. Short windows of silence are common during turn changes. You can already see the structure of a simple supervised training setup using this event set: We can identify occurrences of the 5 events defined above in short sections of two-participant audio conversations. We can label those occurrences. We can train a classifier to predict event occurrences using the labeled data. Problem Setup Prediction Task We'll discuss the prediction task first as it will make understanding the training data structure and training model easier. Here is a graphical representation of the prediction task setup - bear in mind that the diagram is not drawn to scale from a time perspective: Figure 3 - Prediction Task (FinVolution, "Teach AI When to Speak Competition", 2026) Figure 3 - Prediction Task (FinVolution, "Teach AI When to Speak Competition", 2026) Training audio is sliced into non-overlapping 30-second ("s") context windows. Each context window is followed by a 2s prediction window. The training model learns to predict which conversational events, as defined in Table 1 above, occur within each prediction window, given the features within each corresponding context window. Each prediction window is “chunked” as a set of twenty-five 80-millisecond ("ms") audio chunks, and a prediction is made for each individual chunk. In other words, the model outputs a prediction (one of the 5 event labels) for each of the 25 individual chunks, creating an event map of the next two seconds of conversation.Training Data Structure Each sample in the training data, which includes 1,000 total samples, is essentially three separate files: File Description audio/.wav Two-person conversation audio text/.json Text transcript of the conversation audio labels/.npy Conversational event labels C, T, BC, I, NA applied temporally to the conversation It should be obvious that this training data structure supports multimodal modeling where audio, text, and label features can be extracted from each context window. This approach shows you how you could structure your own training data for other languages. I'll expand on this topic further in the Conclusion and Your Own Implementation section.Training Model We can establish an analogy between the multimodal training data and the multisensory experience of human participants during a conversation. For example, the audio, text, and label data features can be mapped to: Listening: How is the speaker speaking - e.g. tone, pauses, etc.? Semantic processing: What is the speaker saying? What is the meaning of his/her words? Dialogue processing: What is the rhythm of the conversation? For example, what is the pace of turns? The baseline training model provided by the competition takes advantage of the multimodal nature of the training data to "behave" in a similar way like a human being - it processes the different types of available data - audio, text, and event labels - simultaneously for a given conversation. It extracts these multimodal features, combines them into a single feature vector, and passes them through a model head that makes the actual predictions. The training model is further characterized by tail-awareness: it pays particular attention to what happened right before an event (e.g. a turn) versus what happened several seconds in the past. If this doesn't make sense to you, consider that the immediate cues for a turn or interruption happen just moments before the actual event. So, assigning more importance to the final seconds of the 30-second context window yields much better predictions than treating the whole 30 seconds equally.Naturally, the audio, text, and label encoders must be modeled. Many winning competition submissions used Whisper with attention pooling to assign higher weights to the tail of each audio sequence. Another approach, which you are likely already familiar with, converts each audio recording to a mel-spectrogram and then passes each mel-spectrogram through a convolutional neural network ("CNN") for feature extraction. Text encoding can be handled by a pre-trained language model such as BERT. A custom neural network can be designed to process the sequence of event labels in each context window. The competition's baseline model uses a two-branch approach where the first branch processes the entire event sequence and the second branch processes those events in the tail of each context window. The baseline model also employs handcrafted features that are largely derived from a statistical analysis of the events in each context window.As hinted above, feature vectors are concatenated and then passed to a final multi-layer neural network which makes the actual predictions.Conclusion and Your Own Implementation To reiterate, the objective of this article was to provide a conceptual understanding versus actual data and training/implementation scripts. At this point, you hopefully have a solid base from which you can apply the concepts discussed above toward your own experiments in Chinese or other languages. You can create you own custom datasets that mimic the structure of the competition dataset by starting with an audio dataset and writing custom scripts to transcribe and label the data. For example, there are conversational speech corpora available for many different languages, such as the famousSwitchboard-1 Release 2 corpus for English. WhisperX can be used to transcribe and diarize each audio example. A simple classifier can ingest the audio, transcription, and diarization data to temporally label each audio recording using the events defined earlier. You can define the "rules" that indicate a given event, e.g.: Silence: No detected speech for 500 or more ms. Backchannel: The person who is not speaking utters a filler word from a defined set, e.g. ["uh-huh", "uh huh", "mm-hmm", "mm hmm", "mhm", "yeah", "yes", "right", "okay", "ok", "oh", "wow", "sure"]. Interruption: Diarization data demonstrates that the person who was not speaking started speaking before the other speaker finished. etc. When designing your loss function, consider that you will likely need to account for class imbalance as, for example, continuations and silences are more likely to dominate the dataset compared to interruptions.The framework used with the FinVolution Teach AI to Speak competition is relatively simple in its description, but powerful in its application and extensibility. Happy building!