Speak the Works and Speak the Words Set issue

Submitted by Jeroen van Dinther on Wed, 12/14/2022 - 12:23
Forums: 
Bugs in content types

Hi,

There appears to be an issue with the voice recording functionality in Speak the Words and Speak the Words Set when used on iPhone.
Based on what I have tested so far, the voice recording functionality works fine when using the Audio Recorder content type, even on the iPhone or in Safari on a MacBook Air. On Windows laptops or an Android mobile phones everything appears to be working fine in the Chrome and Edge browser.

I currently presume, that the Audio Recorder uses a different technology to control the microphone than the Speak the Words and Speak the Words Set control type, or that the problem lies in the speech to text API. The issue must reside in those two content types.
The result is that the microphone button responds to pressing it, but no sound appears to be processed.

Is there any chance someone can look into the issue and solve it?
Your help and efforts are highly appreciated.

Best regards, Jeroen

Summary: 
Speak the Words and Speak the Words Set issue with microphone
BV52's picture

BV52

Wed, 12/14/2022 - 16:42

Hi Jeroen,

Unlike Audio Recorder, Speak the Words does not record your voice, instead it uses a speech recognition software called Annyang. Currently this software is only fully supported in Chrome, we've seen some minor issues in Edge like it adds a period at the end but that can be resolved by just adding alternatives. In the case of iOS even when you use Chrome browser it may not work properly, the reason for this is that Chrome in iOS uses Safari's webkit. This means that Chrome in iOS devices is very similar to it being a skin of Safari.

-BV

otacke's picture

otacke

Wed, 12/14/2022 - 19:13

Sorry BV,

Annyang is no speech recognition software but merely a mini framework for the browser's implementation of  the Web Speech API (https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API). It is the browser that is responsible for supporting that API or not. Each browser implements this API differently (e. g. Chrome using remote Google cloud services), Safari probably using Siri infrastructure (yes, Safari, hold on), etc.). They may all deliver different results.

At the bottom of the web page that I linked to is a list of browsers/browser versions that support the Web Speech API. Safari was late to the party as always, so only newer versions of Safari support the API.

Best,

Oliver 

BV52's picture

BV52

Thu, 12/15/2022 - 17:41

Thanks for the clarfication Oliver!

Jeroen van Dinther

Tue, 04/25/2023 - 12:39

Hi Oliver an BV,

With Safari on iPhone and iPad I have not been able to use this functionality. Since a few weeks ago, after a Chrome update for iOS it works, but with a delay. On iPhone I experience a 5 second delay after speaking, but the text is recognized. On the iPad I experience a 60 seconds delay after speaking, but also there the text is recognized and processed properly. 
Since it started to work, there already have been 2 or 3 new updates to Chrome, but the delays are still the same. I am hoping that Google and Apple can figure out what causes these delays soon. (Probably they forgot to remove a few WAIT commands. lol). Fingers crossed here, because on laptops (iOS and Windows) and on Android phones it works like a charm. It would be great if we can have similar performance levels on iPhone and iPad.

Kind regards, Jeroen.

otacke's picture

otacke

Tue, 04/25/2023 - 22:04

Hi Jeroen!

Please tell Google and Apple in particular. There's nothing that H5P Group could do here.

Best,
Oliver

Jeroen van Dinther

Sat, 09/09/2023 - 19:00

Hi Oliver,

Is there something that H5P group can do to set the recognition.continuous and recognition.interimResults flags/parameters?
Or is that something we should be able to do ourselves?

I have the impression, that despite using HTTPS, where recognition.continuous should default to false, it is using true as the setting on iPad and iPhone.
In the demo from Google at https://www.google.com/intl/en/chrome/demos/speech.html both recognition.continuous and recognition.interimResults are set to true and this demo works on the iPhone and iPad. 
I would like to set if we can get it to work with recognition.continuous set to false and then determine what setting works best for recognition.interimResults. That probably depends on how fast you want answers to be evaluated and how much time you want to allow between two parts of a spoken sentence.

Best regards,
Jeroen.

Jeroen van Dinther

Tue, 04/25/2023 - 16:53

Hi Oliver an BV,

With Safari on iPhone and iPad I have not been able to use this functionality. Since a few weeks ago, after a Chrome update for iOS it works, but with a delay. On iPhone I experience a 5 second delay after speaking, but the text is recognized. On the iPad I experience a 60 seconds delay after speaking, but also there the text is recognized and processed properly. 
Since it started to work, there already have been 2 or 3 new updates to Chrome, but the delays are still the same. I am hoping that Google and Apple can figure out what causes these delays soon. (Probably they forgot to remove a few WAIT commands. lol). Fingers crossed here, because on laptops (iOS and Windows) and on Android phones it works like a charm. It would be great if we can have similar performance levels on iPhone and iPad.

Kind regards, Jeroen.

otacke's picture

otacke

Sun, 09/10/2023 - 13:14

I don't think I can answer on behalf of H5P Group.

Just from a technical perspective: The annyang library that SpeakTheWords is using allows to set the `continuous` flag, and even though `interimResults` are not supported directly, one can supposedly find workarounds. Or one could remove the annyang dependency and use the WebSpeechAPI directly.

Best,

Oliver