Monday, May 9, 2011

Text to Speech Plugin for PhoneGap Android

On Saturday I participated in the Droidhack put on by Ottawa Android downtown at the Code Factory. The Droidhack is an opportunity for Android developers to get together to work on open source Android projects. Luckily, my day job is working on an open source Android project but the Droidhack gave me the opportunity to work on a plugin that I've wanted to do for awhile, text to speech.

At a former job I developed telephony applications that took advantage of speech recognition and text to speech. Since android has built in text to speech functionality I figured I would expose the functionality to PhoneGap developers via a plugin. Maybe some other folks can take advantage of this plugin and provide more accessible Android applications.

In order to use the TTS plugin wait until you get the deviceready event and then call:
window.plugins.tts.startup(startupWin, fail);
This will start the TTS service on your device. However, the service will take a bit of time to start so the startup success callback is first called with a value of TTS.INITIALIZING which tells you that the service is initializing. Once the service is completely started the success callback is executed again with a value of TTS.STARTED which means we are ready to synthesize text.
function startupWin(result) {
 // When result is equal to STARTED we are ready to play
 if (result == TTS.STARTED) {
  //Ready to go
 }
}
To have your device play some text you simply call the speak method:
window.plugins.tts.speak("The text to speech service is ready");
If you want to have some silence between utterances you can call the silence method:
window.plugins.tts.speak("Let me think.");
window.plugins.tts.silence(2000);
window.plugins.tts.speak("I do not know");
In the above example the TTS service will pause for 2 seconds between the two speak calls. Both the speak and silence methods allow you to provide optional success and error callbacks if you need that kind of information.

Of course all of the above examples assume you are using the English (American) package to speak your utterances. If want to find out what the currently assigned TTS language is you call the getLanguage method:
window.plugins.tts.getLanguage(win, fail);
The success callback has one parameter and that will be a text representation of the Locale currently set for the TTS service.

Now, Android by default supports the following languages for the TTS service English (American), English (UK), French, German, Italian and Spanish. However, it is not guaranteed that each package will be installed on your device. So before you set the TTS language you will want to check to see if it is available by calling the isLanguageAvailable method and in the success callback then you set the TTS language you want to use by executing the setLanguage method:
window.plugins.tts.isLanguageAvailable("fr", function() {
   window.plugins.tts.setLanguage("fr");
   }, fail);
And of course if you no longer need the TTS service than call the shutdown method to free up the resources:
window.plugins.tts.shutdown(win, fail);
If you don't call shutdown on the TTS service the service will be shutdown when your device exits.

The plugin is available at the Android PhoneGap Plugin repository. Please give it a download, try it out and let me know what you think.

The following listing shows a full example of using the TTS Plugin:

65 comments:

  1. Hi,

    Thanks for sharing this cool plugin. What do you thin about speech recognition? Is it possible to extend your plugin this way? I'm inexperienced with the native Android API so any suggestion is welcome :)

    Regards,
    Chris

    ReplyDelete
  2. @Chris79 Speech recognition is possible but I will probably create a whole new plugin to support it.

    ReplyDelete
  3. @simon... i cant find tts.js..where i can download that.?? .. did you create new plugin for "text to speech". it would be great if you release that. Thanks in advance

    ReplyDelete
  4. Am trying to find tts for my phonegap application. i found your plugin and try to see the demo in my local. i cant able to find the tts.js. i added phonegap.js and the speech.html. if you provide that it would be great for me to work.
    Thanks in advance

    ReplyDelete
  5. Hi, Simon.This seems like a great plugin. Unfortunately I have been unable to get it to work even after following the instructions on github. This should work fine in phonegap 1.4.1 shouldnt it? Also any phone running android 1.6 or above should be able to run the plugin correct?

    ReplyDelete
  6. @denhosi1

    It should work fine on PhoneGap 1.4.1 but you'll need Android 2.1 or better.

    ReplyDelete
  7. For people using Phonegap 1.5, I let eclipse edit line #69 in the tts.java file from mTts = new TextToSpeech(ctx, this); to mTts = new TextToSpeech((Context) ctx, this);. I tried to learn what (context) means exactly, but it was a very abstract concept. I think it is similar to using this in a function but in a broader scale Everything is working well now!

    Thank you so much for writing this library, this will allow me to make a better art application, and for others to make more accessible/safer/entertaining apps. Thanks!

    ReplyDelete
  8. @gregcoleinfo

    Actually if you are using 1.5 you should take a look at the change I pushed up to Github yesterday. Instead of casting ctx to a Context you call ctx.getContext().

    And finally, you're welcome.

    ReplyDelete
  9. Hi, Simon.This seems like a great plugin. Unfortunately I have been unable to get it to work even after following the instructions on github. TTS.java is a java class or is an android activity. How do i link TTS.java to the HTML. Sorry i just started using phonegap, so i might asked a stupid question

    ReplyDelete
  10. @BK-Stalker

    TTS.java is a Java source file that gets compiled into a class when you build your app. Check out:

    https://github.com/phonegap/phonegap-plugins/tree/master/Android/TTS

    For installation instructions into your project.

    ReplyDelete
  11. Hi Simon,

    I want to highlight the text with speech recognition.Suppose I have a text file so i want to highlight the text which is reading by speech recognition like if it reads Hello so it should highlight Hello.
    I hope you got my point.
    Can you help me with this please.

    Thanks A lot.

    ReplyDelete
  12. @Sarika

    Yeah, that shouldn't be too hard to do. You'd need to break it down so that each word is sent to the speak command and as you do that change the CSS of the element in the web view to show the highlight.

    ReplyDelete
  13. Hi Simon,

    Great plugin.

    FYI: in my testing on a phone with the "Pico TTS" speech engine installed and "English (United Kingdom)" chosen as the language, your plugin reports "en_GBR" (with an "R") as the result of getLanguage().

    Using isLanguageAvailable(), it finds three types of English: en_US, en_GB, and en_GBR. However, en_GB has exactly the same voice as en_US (i.e., with an American accent). Only en_GBR sounds British.

    Keep up the good work!

    ReplyDelete
  14. Hi Simon, I have three important questions:
    The TTS plugin works offline?

    Is posible to install other languages, like Portuguese?

    The response is as fast as lowlatency plugin, or is very slow in comparison?

    Thanks as always!

    ReplyDelete
  15. @nicoprofe

    Yes

    Yes, you can install additional languages. I should update the plugin to use the ACTION_INSTALL_TTS_DATA which takes you to the market to get the new language.
    http://android-developers.blogspot.com/2009/09/introduction-to-text-to-speech-in.html

    I don't know.

    ReplyDelete
  16. Hi Simon,

    The plugin works great. I have 2 questions :

    1. If i want to stop the plugin from speaking, how can i do add. Is there a command for stopping the speech.

    2. Are you planning to make TTS for iOS devices?

    ReplyDelete
  17. @Vishal

    Yes, you can call stop to cancel the playback or interrupt which will stop it and start playing new text you pass in. Check out the tts.js file for all the methods.

    ReplyDelete
  18. Hi Simon, I am using your wonderful plugin in one project.
    Is possible to get a better voice that the one that comes with android (pico TTS)? For example something with the quality of the voice commands app that come with android devices...
    Thanks!

    ReplyDelete
  19. @nicoprofe

    Yes, it is possible to install extra voices and even different TTS engines on Android from version 2.2 or later.

    https://support.google.com/android/bin/answer.py?hl=en&answer=168586#1104525

    I still have that "ACTION_INSTALL_TTS_DATA" on my to do list.

    ReplyDelete
  20. Hi, do you know if voice engines like (Ivonna, At&t, iSpeech, DragonMobile, etc) are Phonegap and TTS plugin friendly?

    About the ACTION_INSTALL_TTS_DATA, I am a kind of javascript developer with no java skills. I'll wait and maybe, if I invite you a coofee :)

    Thanks again Simon!

    ReplyDelete
  21. Hi Simon,

    very nice plugin of you!

    But i have one question:
    if i try let the app read a paragraph with the id="asd" and call this with
    function speak() {
    window.plugins.tts.speak(document.getElementById('muh').value);
    }
    i only get null, the same happens, if i try to let javascript write the text before the speak function.

    Can you tell me how to let the plugin some text, would be very awesome.

    Thanks.

    Regards,
    Alex

    ReplyDelete
  22. @Alex 921

    Can you get the plugin to read any static text? If you can't then the plugin is not installed correctly. My next step would be for you to check for problems in "adb logcat".

    ReplyDelete
  23. @nicoprofe

    I put it on my to do list for when I updated the TTS plugin.

    ReplyDelete
  24. Hi Simon.... i used your TTs plugin but TTS.java i am getting error only in this line [ mTts = new TextToSpeech(cordova.getActivity().getApplicationContext(), this);] which says cordova cannot be resolved.. i am using phonegap-1.3.0.js and phonegap-1.3.0.jar...
    can you please tell me how to correct it....

    ReplyDelete
  25. Hi simon....finally i found out the issue in last post of mine...Now code works fine but when i entered in something in editbox and clicking speak it doesn't speak....can u tell me why....any solution....

    ReplyDelete
  26. @shamsheer

    I'm sure your first issue was caused by the fact you were trying to use the updated plugin with an older version of PhoneGap.

    As for your second problem, what do you see in "adb logcat"?

    ReplyDelete
  27. Hi @Simon....thanks for your god response....In Logcat i am getting like this "No keyboard for id 0" and "Using default keymap: /system/usr/keychars/qwerty.kcm.bin".....I am not getting idea how to solve it...

    ReplyDelete
  28. Hi Simon.. I am developing Phonegap iOS apps by using Xcode 4.5.2. Can you tell me how the TTS plugin add in the Xcode. Thanks in advance

    ReplyDelete
  29. can we have chances to change the voice setting , this voice not clear

    ReplyDelete
  30. Hi Simon.....Finally i got solution .Now it is working fantastically....super cool awesome plugin. My mistake was that i used old version of TTS plugin.... when i use new version TTS plugin. its works clearly..... thanks for your help and your plugin tutorial......

    ReplyDelete
  31. @Test Apps

    You can adjust the speed and pitch of the voice. Look at tts.js for the docs on those methods.

    ReplyDelete
  32. @chakri kasam

    I dont' think anyone has posted one for iOS but if you look at this SO answer you'll be able to find out how to begin creating one:

    http://stackoverflow.com/a/12465829/41679

    ReplyDelete
  33. Hi

    This plugin works really well, but so far i have only been able to test it on english, My dropdown box does not get populated with languages and i am not sure how to make this appear. Can you please help.

    ReplyDelete
  34. @Hayden Sookchand

    That's weird. You can go to the Play store and install more languages though.

    ReplyDelete
  35. Thanks for discussing this awesome plug-in. What do you slim about conversation recognition? Is it possible to boost your plug-in this way? I'm unskilled with local Android operating system API so any . I heard about some tts online that they also have some cool features too.

    ReplyDelete
  36. @Willard Catalan

    For speech recognition use this plugin:

    https://github.com/phonegap/phonegap-plugins/tree/master/Android/SpeechRecognizer

    ReplyDelete
  37. Great plugin. One question. I am calling this on a setInterval to speak an array of items in turn. But I need to know if TTS is still speaking before I interrupt it with the next item. Is it possible to know when the TTS engine has finished speaking?

    ReplyDelete
  38. @John Blessing

    Well the speak method has a success callback which will be invoked when the text is done being spoken. What you should be doing instead of setTimeout is to push all your text into an array then each call of the success callback pops text off the array and reads it to the user.

    ReplyDelete
  39. Sorry to bug you. I have just downloade v 2.2 of the tts plugin.

    I am now calling the speak and specifying a success/fail function, but I never see the success function entered. My code:

    ...
    window.plugins.tts.speak("This is a long piece of text ",ttsSuccess,ttsFail);
    ...

    function ttsSuccess(result) {
    //called after speech has finished
    console.log("ttsSuccess " + result);
    }
    function ttsFail(result) {
    console.log("Error = " + result);
    }


    Any help would be appreciated.

    ReplyDelete
  40. @John Blessing

    Yeah, I see that too. It is an open issue on the plugin. Working on it when I can.

    ReplyDelete
  41. Hello Simon,

    is there a way to create a stop during the reading without shutting down the whole tts service?

    Thanks for your answer.

    Regards
    Ales

    ReplyDelete
  42. @Alex 921

    Call stop to stop any playing tts or interrupt to stop what is currently being played and replace it with new text.

    ReplyDelete
  43. @John Blessing

    Bug has been fixed and code pushed to GitHub.

    ReplyDelete
  44. Fantastic. Works great now. Thanks.

    ReplyDelete
  45. Actually...

    Success function is not reached when speaking silence


    window.plugins.tts.speak(10000ttsSuccess); works fine, but

    window.plugins.tts.silence(10000, ttsSuccess); never sees triggers ttsSuccess

    Thanks in advance.

    ReplyDelete
  46. @John Blessing

    Can you raise an issue on the TTS github repo for me? That way I won't forget to fix it.

    ReplyDelete
  47. Hi Simon,

    I am running Cordova 1.5. I tried to use the version 2.2 TTS plugin as per https://github.com/macdonst/TTS/tree/master/2.2.0 following the instructions at https://github.com/macdonst/TTS/blob/master/README.md to add the plugin to my project.

    Eclipse is reporting many errors in TTS.java:
    - CallbackContext cannot be resolved to a type
    - cordova cannot be resolved
    - CordovaPlugin cannot be resolved to a type
    - The import org.apache.cordova.api.CallbackContext cannot be resolved
    - The import org.apache.cordova.api.CordovaPlugin cannot be resolved

    Is TTS version 2.2 supposed to be compatible with Cordova 1.5? If not, which version of TTS should I use with Cordova 1.5.

    Thanks,
    David

    ReplyDelete
  48. @David Wong

    You are using the wrong version. The 2.2.0 directory includes a Plugin that works from Cordova 2.2.0 and higher. If you are using 1.5.0 then try the 1.8.1 directory.

    ReplyDelete
  49. Thanks Simon for your fast response. I got errors with TTS 1.8.1 and Cordova 1.5. I will try the other lineup of TTS and Cordova. I appreciate your help with the information.

    ReplyDelete
  50. Hi Simon,

    I tried Cordova 2.0.0 with TTS 2.0 as well as Cordova 2.2.0 with TTS 2.2. In both cases, window.plugins.tts.speak("The text to speech service is ready") produces no speech on my Android 2.2.1 smartphone. getLanguage() succeeds with a null string in the callback result. isLanguageAvailable() for "en_US" succeeds and a second call to getLanguage() succeeds with a null string in the callback result.

    I am running the code as in your example. The window.plugins.tts.startup() succeeds with two invokations of the callback with result 1 and 2 (TTS.STARTED).

    Do you have any thoughts on what might be wrong?

    Thanks,
    David

    ReplyDelete
  51. Hi Simon,

    I posted my last message a little too soon! The PhoneGap TTS plugin works after I installed Android's speech to text data (SpeechSynthesis Data Installer) from Google Play.

    David

    ReplyDelete
  52. Hi Simon,

    I am using Cordova 2.2.0 with TTS 2.2. When I launch my app, my index.html calls window.plugins.tts.startup() after device is ready - I get TTS.STARTED - window.plugins.tts.speak() works. When I transition from my base index.html to a second help.html page, and transition back to index.html, I do the same window.plugins.tts.startup() after device is ready - now I do not get TTS.STARTED - fortunately window.plugins.tts.speak() continues to work.

    When I transition from index.html to help.html, I added a call to window.plugins.tts.shutdown() which succeeds. Upon transition from help.html to index.html, I no longer get TTS.STARTED and window.plugins.tts.speak() no longer utters any sound.

    Is there a need to call shutdown() in the way I've described above? Are resources being allocated but not released by not calling shutdown()? However, if I call shutdown() on transition from index.html to help.html, speak() no longer works after re-entry to index.html.

    Thanks,
    David

    ReplyDelete
  53. @David Wong

    Since you are going to continue using the TTS service when you get back to index.html I wouldn't bother calling shutdown.

    ReplyDelete
  54. Thanks for your information Simon. The TTS plugin is working very well in a new app I am developing. Since I target my apps to Android as well as iOS, I wish for a similar Cordova plugin for the latter. Are you aware of any good options for iOS?

    ReplyDelete
  55. Thanks for great plugin,This plugin works with SpeechSynthesis Data Installer from android market

    Is there any way to do it without the use of SpeechSynthesis Data Installer..

    ReplyDelete
  56. @rajan singh

    Yeah, if your device does not come pre-installed with Google TTS then you will need to install a 3rd party TTS.

    ReplyDelete
  57. @David Wong

    Apple does not provide an SDK for TTS. You'd have to use a third party solution and wrap it in a plugin like:

    http://www.speech.cs.cmu.edu/flite/

    ReplyDelete
  58. Thanks for the lead, Simon. I'll investigate.

    ReplyDelete
  59. For version 3.0 of cordova , the package name seems to have changed.
    from
    import org.apache.cordova.api.CallbackContext;
    import org.apache.cordova.api.CordovaPlugin;
    import org.apache.cordova.api.PluginResult;

    to
    import org.apache.cordova.*;

    after making this change I am able to compile with cordova 3.0

    ReplyDelete
  60. @Bhavesh Patel

    Use this repo for 3.0.0 support.

    https://github.com/macdonst/SpeechSynthesisPlugin

    I will be doing a blog post on it soon.

    ReplyDelete
  61. Hello ...

    Is there someone that might have a link to a simple test app (in phonegap) that uses the above tts code ...I could download and test on my android Device ...

    Thank you in advance ...

    Michael T.

    ReplyDelete
  62. @Michael

    This plugin is very out of date. I've switched over to basing things off of the W3C Speech Synthesis spec. Here is a demo repo:

    https://github.com/macdonst/ttsdemo

    ReplyDelete
  63. How to get end of speak event.
    I want to speak next string after end of currently playing one.
    is there any event which give me end event?

    ReplyDelete
  64. End of speak event for auto play next item after complete current one
    this plugin can provide end event for current playing item is end to speak.

    ReplyDelete