We control the standard Sailfish OS player using voice commands
- Tutorial
Many people know and use such features of the Android operating system as Google Now and Google Assistant, which allow not only to get useful information and search on the Internet in time, but also to control the device using voice commands. Unfortunately, Sailfish OS (an operating system developed by the Finnish company Jolla and the Russian company Open Mobile Platform) does not provide such an opportunity “out of the box”. As a result, it was decided to make up for the lack of these amenities on their own. One of the functions of the developed solution is the ability to control the music player using voice commands, the technical side of which will be discussed in this article.
To implement voice recognition and execution, you need to go through four simple steps:
It is assumed that, for a better understanding of the material, the reader already has a basic knowledge of C ++, JavaScript, Qt, QML and Linux and got acquainted with an example of their interaction within the framework of Sailfish OS . It may also be useful to get acquainted with a lecture on related topics held as part of the Sailfish OS Summer School in Innopolis in the summer of 2016, and other articles on development for this platform that have already been published on Habré.
Let us examine a simple example limited to five functions:
To start a new playback, you need to check the availability of an open instance of the player (if necessary, create) and start playing music in random order. To activate, we will use the “Turn on music” command.
In the case of resuming and pausing playback, you need to check the status of the player and, if possible, start playback or pause it. To resume playback, we will use the “Play” command; to pause, use the “Pause” and “Stop” commands.
In the case of navigating through the compositions, the above principle of checking the status of the audio player applies. To activate forward navigation, use the “Forward”, “Next” and “Next” commands; To activate backward navigation, use the “Back” and “Previous” commands.
The recognition process of voice commands is divided into three stages:
First you need to create a user interface for capturing a voice command. In order to simplify the example, we will start and end the recording by pressing the button, since the implementation of the process of detecting the beginning and end of a voice command deserves separate material.
From the code presented above, it can be seen that the button uses standard sizes and standard icons (an interesting feature of Sailfish OS to unify application interfaces) and has two states. In the first state, when recording is not performed, after pressing the button, recording of a voice command begins. In the second state, when the command recording is active, after pressing the button, the recording stops and voice recognition starts.
To record speech, we will use the QAudioRecorder class , which provides a high-level interface for controlling the input audio stream, as well as QAudioEncoderSettings for adjusting the recording process.
It indicates that the recording of the command will be in wav format in normal quality, and methods for starting and ending the recording, for obtaining the storage location of the audio file and the status of the recording process are also determined.
The Yandex SpeechKit Cloud service will be used to broadcast the audio file to text . All that is required to start working with it is to get a token in the developer's office . The service documentation is quite detailed, so we will dwell only on private moments.
The first step is to transfer the recorded command to the server.
Here, a POST request is generated to the Yandex server, in which the received token is transmitted, a unique device ID (in this case, the MAC address of the WiFi module is used) and the type of request (“queries” are used here, since voice interaction with the device is most often used short and precise commands). The request headers indicate the format of the audio file and its size, in the body - directly the content. After sending the request to the server, the file is deleted as unnecessary.
As a response, the SpeechKit Cloud server returns XML with recognition options and a degree of confidence in them. We use standard Qt tools to highlight the required information.
Here, the received answer is successively reviewed and, for variant tags, recognition accuracy indicators are checked. If the new option is correct, then it is saved, and scanning continues. At the end of viewing the response, a signal is sent with the selected command text.
Finally, it remains to identify the team. At the end of the YandexSpeechKitHelper :: _ parseResponce method , as mentioned above, a gotResponce signal containing the command text is sent . Next, you need to process it in the QML code of the program.
This uses the Connections element to process the incoming signal and compare the recognized command with the voice command patterns defined previously.
If the audio player is open, then it is possible to interact with it through the standard DBus interface , inherited from the big linux brother. With it, you can navigate the playlist, start or pause playback. This is done using the DBMLInterface QML element .
Using this element, the DBus interface of a standard audio player is used by defining four basic functions. The undefined parameter of the call function is passed if the DBus method does not accept arguments.
It is worth noting that in order to go to the previous song, the Previous method is called twice, since a single call to it will play the current song from the beginning.
There is nothing complicated in managing an already running player. However, if there is a desire to start playing music when it is closed, a problem arises, because, by default, the standard player’s launch functionality while playing the entire collection is not provided.
But do not forget that Sailfish OS is an open source operating system available for free modification. As a result of this, the problem can be solved in two stages:
The standard audio player, in addition to the org.mpris.MediaPlayer2.Player interface , provides the com.jolla.mediaplayer.ui interface defined in the /usr/share/jolla-mediaplayer/mediaplayer.qml file . From this it follows that it is possible to modify this file by adding the function we need.
Here, the DBusAdaptor element used to provide the DBus interface was modified by adding the shuffleAndPlay method . It uses the standard player functionality to start playing all songs in random order, provided by the c om.jolla.mediaplayer module , and the current playback queue is brought to the fore.
As part of the example, for simplicity, a simple modification of the system file was performed. However, when distributing the program, such changes must be made out in the form of patches, using the appropriate instructions .
Now, from the program being developed, it is necessary to turn to the new method. This is done using an already familiar element.DBusInterface , which connects to the service defined above and calls the function added to the player.
Finally, the last thing left is to launch the audio player if it is closed. Conventionally, a task can be divided into two stages:
It can be seen from the code of the presented function that at the first stage, the availability of the necessary DBus service is checked. If it is registered in the system, the function terminates and the transition to starting playback starts. If the service is not found, then a new instance of the audio player is created using QProcess , with the expectation of its full launch. In the second part of the function, using QDBusInterface , the flag of the end of scanning the music collection on the device is checked.
It should be noted that two additional changes were made to the /usr/share/jolla-mediaplayer/mediaplayer.qml file to check the collection scan flag .
Firstly, the GriloTrackerModel element provided by the module has been modifiedcom.jolla.mediaplayer , by adding a scan end flag.
Secondly, it has added another feature available through a DBus interface- com.jolla.mediaplayer.ui , returning the value of the state flag scan audio files collection.
The last element of the example is a voice message about the wrong command. To do this, use the Yandex SpeechKit Cloud speech synthesis service.
Here, an Audio object was created to reproduce the generated speech and the generateErrorMessage function was declared to generate a request to the Yandex server and start playback. The request contains the following parameters:
This article discusses a simple example of controlling music playback in a standard Sailfish OS audio player using voice commands; Basic knowledge about speech recognition and synthesis using Yandex SpeechKit Cloud using Qt tools was obtained and repeated, as well as the principles of how programs interact with each other in Sailfish OS. This material can serve as a starting point for deeper research and experiments in this operating system.
An example of the above code can be seen in the video:
Author: Pyotr Vytovtov
To implement voice recognition and execution, you need to go through four simple steps:
- develop a team system
- implement speech recognition
- implement identification and execution of commands,
- add voice feedback.
It is assumed that, for a better understanding of the material, the reader already has a basic knowledge of C ++, JavaScript, Qt, QML and Linux and got acquainted with an example of their interaction within the framework of Sailfish OS . It may also be useful to get acquainted with a lecture on related topics held as part of the Sailfish OS Summer School in Innopolis in the summer of 2016, and other articles on development for this platform that have already been published on Habré.
Team System Development
Let us examine a simple example limited to five functions:
- launching new music playback,
- resume playing music
- pause music
- Skip to the next song
- Go to the previous song.
To start a new playback, you need to check the availability of an open instance of the player (if necessary, create) and start playing music in random order. To activate, we will use the “Turn on music” command.
In the case of resuming and pausing playback, you need to check the status of the player and, if possible, start playback or pause it. To resume playback, we will use the “Play” command; to pause, use the “Pause” and “Stop” commands.
In the case of navigating through the compositions, the above principle of checking the status of the audio player applies. To activate forward navigation, use the “Forward”, “Next” and “Next” commands; To activate backward navigation, use the “Back” and “Previous” commands.
Speech recognition
The recognition process of voice commands is divided into three stages:
- recording a voice command to a file,
- command recognition on the server,
- team identification on the device.
Recording a voice command to a file
First you need to create a user interface for capturing a voice command. In order to simplify the example, we will start and end the recording by pressing the button, since the implementation of the process of detecting the beginning and end of a voice command deserves separate material.
IconButton {
property bool isRecording: false
width: Theme.iconSizeLarge
height: Theme.iconSizeLarge
icon.source: isRecording ? "image://theme/icon-m-search" :
"image://theme/icon-m-mic"
onClicked: {
if (isRecording) {
isRecording = false
recorder.stopRecord()
yandexSpeechKitHelper.recognizeQuery(recorder.getActualLocation())
} else {
isRecording = true
recorder.startRecord()
}
}
}
From the code presented above, it can be seen that the button uses standard sizes and standard icons (an interesting feature of Sailfish OS to unify application interfaces) and has two states. In the first state, when recording is not performed, after pressing the button, recording of a voice command begins. In the second state, when the command recording is active, after pressing the button, the recording stops and voice recognition starts.
To record speech, we will use the QAudioRecorder class , which provides a high-level interface for controlling the input audio stream, as well as QAudioEncoderSettings for adjusting the recording process.
classRecorder :public QObject
{
Q_OBJECT
public:
explicitRecorder(QObject *parent = 0);
Q_INVOKABLE voidstartRecord();
Q_INVOKABLE voidstopRecord();
Q_INVOKABLE QUrl getActualLocation();
Q_INVOKABLE boolisRecording();
private:
QAudioRecorder _audioRecorder;
QAudioEncoderSettings _settings;
bool _recording = false;
};
Recorder::Recorder(QObject *parent) : QObject(parent) {
_settings.setCodec("audio/PCM");
_settings.setQuality(QMultimedia::NormalQuality);
_audioRecorder.setEncodingSettings(_settings);
_audioRecorder.setContainerFormat("wav");
}
void Recorder::startRecord() {
_recording = true;
_audioRecorder.record();
}
void Recorder::stopRecord() {
_recording = false;
_audioRecorder.stop();
}
QUrl Recorder::getActualLocation() {
return _audioRecorder.actualLocation();
}
bool Recorder::isRecording() {
return _recording;
}
It indicates that the recording of the command will be in wav format in normal quality, and methods for starting and ending the recording, for obtaining the storage location of the audio file and the status of the recording process are also determined.
Server command recognition
The Yandex SpeechKit Cloud service will be used to broadcast the audio file to text . All that is required to start working with it is to get a token in the developer's office . The service documentation is quite detailed, so we will dwell only on private moments.
The first step is to transfer the recorded command to the server.
void YandexSpeechKitHelper::recognizeQuery(QString path_to_file) {
QFile *file = new QFile(path_to_file);
if (file->open(QIODevice::ReadOnly)) {
QUrlQuery query;
query.addQueryItem("key", "API_KEY");
query.addQueryItem("uuid", _buildUniqID());
query.addQueryItem("topic", "queries");
QUrl url("https://asr.yandex.net/asr_xml");
url.setQuery(query);
QNetworkRequest request(url);
request.setHeader(QNetworkRequest::ContentTypeHeader, "audio/x-wav");
request.setHeader(QNetworkRequest::ContentLengthHeader, file->size());
_manager->post(request, file->readAll());
file->close();
}
file->remove();
}
Here, a POST request is generated to the Yandex server, in which the received token is transmitted, a unique device ID (in this case, the MAC address of the WiFi module is used) and the type of request (“queries” are used here, since voice interaction with the device is most often used short and precise commands). The request headers indicate the format of the audio file and its size, in the body - directly the content. After sending the request to the server, the file is deleted as unnecessary.
As a response, the SpeechKit Cloud server returns XML with recognition options and a degree of confidence in them. We use standard Qt tools to highlight the required information.
void YandexSpeechKitHelper::_parseResponce(QXmlStreamReader *element) {
double idealConfidence = 0;
QString idealQuery;
while (!element->atEnd()) {
element->readNext();
if (element->tokenType() != QXmlStreamReader::StartElement) continue;
if (element->name() != "variant") continue;
QXmlStreamAttribute attr = element->attributes().at(0);
if (attr.value().toDouble() > idealConfidence) {
idealConfidence = attr.value().toDouble();
element->readNext();
idealQuery = element->text().toString();
}
}
if (element->hasError()) qDebug() << element->errorString();
emit gotResponce(idealQuery);
}
Here, the received answer is successively reviewed and, for variant tags, recognition accuracy indicators are checked. If the new option is correct, then it is saved, and scanning continues. At the end of viewing the response, a signal is sent with the selected command text.
Device identification
Finally, it remains to identify the team. At the end of the YandexSpeechKitHelper :: _ parseResponce method , as mentioned above, a gotResponce signal containing the command text is sent . Next, you need to process it in the QML code of the program.
Connections {
target: yandexSpeechKitHelper
onGotResponce: {
switch (query.toLowerCase()) {
case"включи музыку":
dbusHelper.startMediaplayerIfNeed()
mediaPlayer.shuffleAndPlay()
break;
case"играй":
mediaPlayerControl.play()
break;
case"пауза":
case"стоп":
mediaPlayerControl.pause()
break;
case"вперед":
case"дальше":
case"следующий":
mediaPlayerControl.next()
break;
case"назад":
case"предыдущий":
mediaPlayerControl.previous()
break;
default:
generateErrorMessage(query)
break;
}
}
}
This uses the Connections element to process the incoming signal and compare the recognized command with the voice command patterns defined previously.
Management of a working player
If the audio player is open, then it is possible to interact with it through the standard DBus interface , inherited from the big linux brother. With it, you can navigate the playlist, start or pause playback. This is done using the DBMLInterface QML element .
DBusInterface {
id: mediaPlayerControl
service: "org.mpris.MediaPlayer2.jolla-mediaplayer"
iface: "org.mpris.MediaPlayer2.Player"
path: "/org/mpris/MediaPlayer2"functionplay() {
call("Play", undefined)
}
functionpause() {
call("Pause", undefined)
}
functionnext() {
call("Next", undefined)
}
functionprevious() {
call("Previous", undefined)
call("Previous", undefined)
}
}
Using this element, the DBus interface of a standard audio player is used by defining four basic functions. The undefined parameter of the call function is passed if the DBus method does not accept arguments.
It is worth noting that in order to go to the previous song, the Previous method is called twice, since a single call to it will play the current song from the beginning.
Starting playback from scratch
There is nothing complicated in managing an already running player. However, if there is a desire to start playing music when it is closed, a problem arises, because, by default, the standard player’s launch functionality while playing the entire collection is not provided.
But do not forget that Sailfish OS is an open source operating system available for free modification. As a result of this, the problem can be solved in two stages:
- To expand the functions provided by the player through the DBus-interface;
- Launch the player (if necessary) and start playback immediately after launch.
Extending the functions of a standard audio player
The standard audio player, in addition to the org.mpris.MediaPlayer2.Player interface , provides the com.jolla.mediaplayer.ui interface defined in the /usr/share/jolla-mediaplayer/mediaplayer.qml file . From this it follows that it is possible to modify this file by adding the function we need.
DBusAdaptor {
service: "com.jolla.mediaplayer"
path: "/com/jolla/mediaplayer/ui"
iface: "com.jolla.mediaplayer.ui"functionopenUrl(arg) {
if (arg[0] == undefined) {
returnfalse
}
AudioPlayer.playUrl(Qt.resolvedUrl(arg[0]))
if (!pageStack.currentPage || pageStack.currentPage.objectName !== "PlayQueuePage") {
root.pageStack.push(playQueuePage, {}, PageStackAction.Immediate)
}
activate()
returntrue
}
functionshuffleAndPlay() {
AudioPlayer.shuffleAndPlay(allSongModel, allSongModel.count)
if (!pageStack.currentPage || pageStack.currentPage.objectName !== "PlayQueuePage") {
root.pageStack.push(playQueuePage, {}, PageStackAction.Immediate)
}
activate()
returntrue
}
}
Here, the DBusAdaptor element used to provide the DBus interface was modified by adding the shuffleAndPlay method . It uses the standard player functionality to start playing all songs in random order, provided by the c om.jolla.mediaplayer module , and the current playback queue is brought to the fore.
As part of the example, for simplicity, a simple modification of the system file was performed. However, when distributing the program, such changes must be made out in the form of patches, using the appropriate instructions .
Now, from the program being developed, it is necessary to turn to the new method. This is done using an already familiar element.DBusInterface , which connects to the service defined above and calls the function added to the player.
DBusInterface {
id: mediaPlayer
service: "com.jolla.mediaplayer"
iface: "com.jolla.mediaplayer.ui"
path: "/com/jolla/mediaplayer/ui"functionshuffleAndPlay() {
call("shuffleAndPlay", undefined)
}
}
Player launch if closed
Finally, the last thing left is to launch the audio player if it is closed. Conventionally, a task can be divided into two stages:
- directly launching the player,
- Waiting for a scan of a music collection.
void DBusHelper::startMediaplayerIfNeed() {
QDBusReply<bool> reply =
QDBusConnection::sessionBus().interface()->isServiceRegistered("com.jolla.mediaplayer");
if (!reply.value()) {
QProcess process;
process.start("/bin/bash -c \"jolla-mediaplayer &\"");
process.waitForFinished();
QDBusInterface interface("com.jolla.mediaplayer", "/com/jolla/mediaplayer/ui",
"com.jolla.mediaplayer.ui");
while (true) {
QDBusReply<bool> reply = interface.call("isSongsModelFinished");
if (reply.isValid() && reply.value()) break;
QThread::sleep(1);
}
}
}
It can be seen from the code of the presented function that at the first stage, the availability of the necessary DBus service is checked. If it is registered in the system, the function terminates and the transition to starting playback starts. If the service is not found, then a new instance of the audio player is created using QProcess , with the expectation of its full launch. In the second part of the function, using QDBusInterface , the flag of the end of scanning the music collection on the device is checked.
It should be noted that two additional changes were made to the /usr/share/jolla-mediaplayer/mediaplayer.qml file to check the collection scan flag .
Firstly, the GriloTrackerModel element provided by the module has been modifiedcom.jolla.mediaplayer , by adding a scan end flag.
GriloTrackerModel {
id: allSongModel
property bool isFinished: false
query: {
//: placeholder string for albums without a known name//% "Unknown album"var unknownAlbum = qsTrId("mediaplayer-la-unknown-album")
//: placeholder string to be shown for media without a known artist//% "Unknown artist"var unknownArtist = qsTrId("mediaplayer-la-unknown-artist")
return AudioTrackerHelpers.getSongsQuery("", {"unknownArtist": unknownArtist, "unknownAlbum": unknownAlbum})
}
onFinished: {
isFinished = truevar artList = fetchAlbumArts(3)
if (artList[0]) {
if (!artList[0].url || artList[0].url == "") {
mediaPlayerCover.idleArtist = artList[0].author ? artList[0].author : ""
mediaPlayerCover.idleSong = artList[0].title ? artList[0].title : ""
} else {
mediaPlayerCover.idle.largeAlbumArt = artList[0].url
mediaPlayerCover.idle.leftSmallAlbumArt = artList[1] && artList[1].url ? artList[1].url : ""
mediaPlayerCover.idle.rightSmallAlbumArt = artList[2] && artList[2].url ? artList[2].url : ""
mediaPlayerCover.idle.sourcesReady = true
}
}
}
}
Secondly, it has added another feature available through a DBus interface- com.jolla.mediaplayer.ui , returning the value of the state flag scan audio files collection.
functionisSongsModelFinished() {
return allSongModel.isFinished
}
Error command message
The last element of the example is a voice message about the wrong command. To do this, use the Yandex SpeechKit Cloud speech synthesis service.
Audio { id: audio }
functiongenerateErrorMessage(query) {
var message = "Извините. Команда " + query + " не найдена."
audio.source = "https://tts.voicetech.yandex.net/generate?" +
"text=\"" + message + "\"&" +
"format=mp3&" +
"lang=ru-RU&" +
"speaker=jane&" +
"emotion=good&" +
"key=API_KEY"
audio.play()
}
Here, an Audio object was created to reproduce the generated speech and the generateErrorMessage function was declared to generate a request to the Yandex server and start playback. The request contains the following parameters:
- text - text to synthesize (message about an invalid voice command),
- format - format of the returned file (mp3),
- lang - language of the phrase (Russian),
- speaker - voice acting (female),
- emotion - emotional coloring of the voice (benevolent),
- key - the key received at the beginning of the article.
Conclusion
This article discusses a simple example of controlling music playback in a standard Sailfish OS audio player using voice commands; Basic knowledge about speech recognition and synthesis using Yandex SpeechKit Cloud using Qt tools was obtained and repeated, as well as the principles of how programs interact with each other in Sailfish OS. This material can serve as a starting point for deeper research and experiments in this operating system.
An example of the above code can be seen in the video:
Author: Pyotr Vytovtov
