Monday, May 9, 2011

Text to Speech Plugin for PhoneGap Android

On Saturday I participated in the Droidhack put on by Ottawa Android downtown at the Code Factory. The Droidhack is an opportunity for Android developers to get together to work on open source Android projects. Luckily, my day job is working on an open source Android project but the Droidhack gave me the opportunity to work on a plugin that I've wanted to do for awhile, text to speech.

At a former job I developed telephony applications that took advantage of speech recognition and text to speech. Since android has built in text to speech functionality I figured I would expose the functionality to PhoneGap developers via a plugin. Maybe some other folks can take advantage of this plugin and provide more accessible Android applications.

In order to use the TTS plugin wait until you get the deviceready event and then call:
window.plugins.tts.startup(startupWin, fail);
This will start the TTS service on your device. However, the service will take a bit of time to start so the startup success callback is first called with a value of TTS.INITIALIZING which tells you that the service is initializing. Once the service is completely started the success callback is executed again with a value of TTS.STARTED which means we are ready to synthesize text.
function startupWin(result) {
 // When result is equal to STARTED we are ready to play
 if (result == TTS.STARTED) {
  //Ready to go
 }
}
To have your device play some text you simply call the speak method:
window.plugins.tts.speak("The text to speech service is ready");
If you want to have some silence between utterances you can call the silence method:
window.plugins.tts.speak("Let me think.");
window.plugins.tts.silence(2000);
window.plugins.tts.speak("I do not know");
In the above example the TTS service will pause for 2 seconds between the two speak calls. Both the speak and silence methods allow you to provide optional success and error callbacks if you need that kind of information.

Of course all of the above examples assume you are using the English (American) package to speak your utterances. If want to find out what the currently assigned TTS language is you call the getLanguage method:
window.plugins.tts.getLanguage(win, fail);
The success callback has one parameter and that will be a text representation of the Locale currently set for the TTS service.

Now, Android by default supports the following languages for the TTS service English (American), English (UK), French, German, Italian and Spanish. However, it is not guaranteed that each package will be installed on your device. So before you set the TTS language you will want to check to see if it is available by calling the isLanguageAvailable method and in the success callback then you set the TTS language you want to use by executing the setLanguage method:
window.plugins.tts.isLanguageAvailable("fr", function() {
   window.plugins.tts.setLanguage("fr");
   }, fail);
And of course if you no longer need the TTS service than call the shutdown method to free up the resources:
window.plugins.tts.shutdown(win, fail);
If you don't call shutdown on the TTS service the service will be shutdown when your device exits.

The plugin is available at the Android PhoneGap Plugin repository. Please give it a download, try it out and let me know what you think.

The following listing shows a full example of using the TTS Plugin:

65 comments:

Chris79 said...

Hi,

Thanks for sharing this cool plugin. What do you thin about speech recognition? Is it possible to extend your plugin this way? I'm inexperienced with the native Android API so any suggestion is welcome :)

Regards,
Chris

Simon MacDonald said...

@Chris79 Speech recognition is possible but I will probably create a whole new plugin to support it.

XX said...

@simon... i cant find tts.js..where i can download that.?? .. did you create new plugin for "text to speech". it would be great if you release that. Thanks in advance

XX said...

Am trying to find tts for my phonegap application. i found your plugin and try to see the demo in my local. i cant able to find the tts.js. i added phonegap.js and the speech.html. if you provide that it would be great for me to work.
Thanks in advance

Simon MacDonald said...

@abdulrahman You can get all the code from GitHub. https://github.com/phonegap/phonegap-plugins/tree/master/Android/TTS

denhosi1 said...

Hi, Simon.This seems like a great plugin. Unfortunately I have been unable to get it to work even after following the instructions on github. This should work fine in phonegap 1.4.1 shouldnt it? Also any phone running android 1.6 or above should be able to run the plugin correct?

Simon MacDonald said...

@denhosi1

It should work fine on PhoneGap 1.4.1 but you'll need Android 2.1 or better.

gregcoleinfo said...

For people using Phonegap 1.5, I let eclipse edit line #69 in the tts.java file from mTts = new TextToSpeech(ctx, this); to mTts = new TextToSpeech((Context) ctx, this);. I tried to learn what (context) means exactly, but it was a very abstract concept. I think it is similar to using this in a function but in a broader scale Everything is working well now!

Thank you so much for writing this library, this will allow me to make a better art application, and for others to make more accessible/safer/entertaining apps. Thanks!

Simon MacDonald said...

@gregcoleinfo

Actually if you are using 1.5 you should take a look at the change I pushed up to Github yesterday. Instead of casting ctx to a Context you call ctx.getContext().

And finally, you're welcome.

BK-Stalker said...

Hi, Simon.This seems like a great plugin. Unfortunately I have been unable to get it to work even after following the instructions on github. TTS.java is a java class or is an android activity. How do i link TTS.java to the HTML. Sorry i just started using phonegap, so i might asked a stupid question

Simon MacDonald said...

@BK-Stalker

TTS.java is a Java source file that gets compiled into a class when you build your app. Check out:

https://github.com/phonegap/phonegap-plugins/tree/master/Android/TTS

For installation instructions into your project.

Sarika said...

Hi Simon,

I want to highlight the text with speech recognition.Suppose I have a text file so i want to highlight the text which is reading by speech recognition like if it reads Hello so it should highlight Hello.
I hope you got my point.
Can you help me with this please.

Thanks A lot.

Simon MacDonald said...

@Sarika

Yeah, that shouldn't be too hard to do. You'd need to break it down so that each word is sent to the speak command and as you do that change the CSS of the element in the web view to show the highlight.

Unknown said...

Hi Simon,

Great plugin.

FYI: in my testing on a phone with the "Pico TTS" speech engine installed and "English (United Kingdom)" chosen as the language, your plugin reports "en_GBR" (with an "R") as the result of getLanguage().

Using isLanguageAvailable(), it finds three types of English: en_US, en_GB, and en_GBR. However, en_GB has exactly the same voice as en_US (i.e., with an American accent). Only en_GBR sounds British.

Keep up the good work!

nicoprofe said...

Hi Simon, I have three important questions:
The TTS plugin works offline?

Is posible to install other languages, like Portuguese?

The response is as fast as lowlatency plugin, or is very slow in comparison?

Thanks as always!

Simon MacDonald said...

@nicoprofe

Yes

Yes, you can install additional languages. I should update the plugin to use the ACTION_INSTALL_TTS_DATA which takes you to the market to get the new language.
http://android-developers.blogspot.com/2009/09/introduction-to-text-to-speech-in.html

I don't know.

Anonymous said...

Hi Simon,

The plugin works great. I have 2 questions :

1. If i want to stop the plugin from speaking, how can i do add. Is there a command for stopping the speech.

2. Are you planning to make TTS for iOS devices?

Simon MacDonald said...

@Vishal

Yes, you can call stop to cancel the playback or interrupt which will stop it and start playing new text you pass in. Check out the tts.js file for all the methods.

nicoprofe said...

Hi Simon, I am using your wonderful plugin in one project.
Is possible to get a better voice that the one that comes with android (pico TTS)? For example something with the quality of the voice commands app that come with android devices...
Thanks!

Simon MacDonald said...

@nicoprofe

Yes, it is possible to install extra voices and even different TTS engines on Android from version 2.2 or later.

https://support.google.com/android/bin/answer.py?hl=en&answer=168586#1104525

I still have that "ACTION_INSTALL_TTS_DATA" on my to do list.

nicoprofe said...

Hi, do you know if voice engines like (Ivonna, At&t, iSpeech, DragonMobile, etc) are Phonegap and TTS plugin friendly?

About the ACTION_INSTALL_TTS_DATA, I am a kind of javascript developer with no java skills. I'll wait and maybe, if I invite you a coofee :)

Thanks again Simon!

Alex 921 said...

Hi Simon,

very nice plugin of you!

But i have one question:
if i try let the app read a paragraph with the id="asd" and call this with
function speak() {
window.plugins.tts.speak(document.getElementById('muh').value);
}
i only get null, the same happens, if i try to let javascript write the text before the speak function.

Can you tell me how to let the plugin some text, would be very awesome.

Thanks.

Regards,
Alex

Simon MacDonald said...

@Alex 921

Can you get the plugin to read any static text? If you can't then the plugin is not installed correctly. My next step would be for you to check for problems in "adb logcat".

Simon MacDonald said...

@nicoprofe

I put it on my to do list for when I updated the TTS plugin.

shamsheer said...

Hi Simon.... i used your TTs plugin but TTS.java i am getting error only in this line [ mTts = new TextToSpeech(cordova.getActivity().getApplicationContext(), this);] which says cordova cannot be resolved.. i am using phonegap-1.3.0.js and phonegap-1.3.0.jar...
can you please tell me how to correct it....

shamsheer said...

Hi simon....finally i found out the issue in last post of mine...Now code works fine but when i entered in something in editbox and clicking speak it doesn't speak....can u tell me why....any solution....

Simon MacDonald said...

@shamsheer

I'm sure your first issue was caused by the fact you were trying to use the updated plugin with an older version of PhoneGap.

As for your second problem, what do you see in "adb logcat"?

shamsheer said...

Hi @Simon....thanks for your god response....In Logcat i am getting like this "No keyboard for id 0" and "Using default keymap: /system/usr/keychars/qwerty.kcm.bin".....I am not getting idea how to solve it...

Unknown said...

Hi Simon.. I am developing Phonegap iOS apps by using Xcode 4.5.2. Can you tell me how the TTS plugin add in the Xcode. Thanks in advance

Test Apps said...

can we have chances to change the voice setting , this voice not clear

shamsheer said...

Hi Simon.....Finally i got solution .Now it is working fantastically....super cool awesome plugin. My mistake was that i used old version of TTS plugin.... when i use new version TTS plugin. its works clearly..... thanks for your help and your plugin tutorial......

Simon MacDonald said...

@Test Apps

You can adjust the speed and pitch of the voice. Look at tts.js for the docs on those methods.

Simon MacDonald said...

@chakri kasam

I dont' think anyone has posted one for iOS but if you look at this SO answer you'll be able to find out how to begin creating one:

http://stackoverflow.com/a/12465829/41679

Hayden Sookchand said...

Hi

This plugin works really well, but so far i have only been able to test it on english, My dropdown box does not get populated with languages and i am not sure how to make this appear. Can you please help.

Simon MacDonald said...

@Hayden Sookchand

That's weird. You can go to the Play store and install more languages though.

KirknesS said...

Thanks for discussing this awesome plug-in. What do you slim about conversation recognition? Is it possible to boost your plug-in this way? I'm unskilled with local Android operating system API so any . I heard about some tts online that they also have some cool features too.

Simon MacDonald said...

@Willard Catalan

For speech recognition use this plugin:

https://github.com/phonegap/phonegap-plugins/tree/master/Android/SpeechRecognizer

Unknown said...

Great plugin. One question. I am calling this on a setInterval to speak an array of items in turn. But I need to know if TTS is still speaking before I interrupt it with the next item. Is it possible to know when the TTS engine has finished speaking?

Simon MacDonald said...

@John Blessing

Well the speak method has a success callback which will be invoked when the text is done being spoken. What you should be doing instead of setTimeout is to push all your text into an array then each call of the success callback pops text off the array and reads it to the user.

Unknown said...

Sorry to bug you. I have just downloade v 2.2 of the tts plugin.

I am now calling the speak and specifying a success/fail function, but I never see the success function entered. My code:

...
window.plugins.tts.speak("This is a long piece of text ",ttsSuccess,ttsFail);
...

function ttsSuccess(result) {
//called after speech has finished
console.log("ttsSuccess " + result);
}
function ttsFail(result) {
console.log("Error = " + result);
}


Any help would be appreciated.

Simon MacDonald said...

@John Blessing

Yeah, I see that too. It is an open issue on the plugin. Working on it when I can.

Alex 921 said...

Hello Simon,

is there a way to create a stop during the reading without shutting down the whole tts service?

Thanks for your answer.

Regards
Ales

Simon MacDonald said...

@Alex 921

Call stop to stop any playing tts or interrupt to stop what is currently being played and replace it with new text.

Simon MacDonald said...

@John Blessing

Bug has been fixed and code pushed to GitHub.

Unknown said...

Fantastic. Works great now. Thanks.

Unknown said...

Actually...

Success function is not reached when speaking silence


window.plugins.tts.speak(10000ttsSuccess); works fine, but

window.plugins.tts.silence(10000, ttsSuccess); never sees triggers ttsSuccess

Thanks in advance.

Simon MacDonald said...

@John Blessing

Can you raise an issue on the TTS github repo for me? That way I won't forget to fix it.

Unknown said...

Hi Simon,

I am running Cordova 1.5. I tried to use the version 2.2 TTS plugin as per https://github.com/macdonst/TTS/tree/master/2.2.0 following the instructions at https://github.com/macdonst/TTS/blob/master/README.md to add the plugin to my project.

Eclipse is reporting many errors in TTS.java:
- CallbackContext cannot be resolved to a type
- cordova cannot be resolved
- CordovaPlugin cannot be resolved to a type
- The import org.apache.cordova.api.CallbackContext cannot be resolved
- The import org.apache.cordova.api.CordovaPlugin cannot be resolved

Is TTS version 2.2 supposed to be compatible with Cordova 1.5? If not, which version of TTS should I use with Cordova 1.5.

Thanks,
David

Simon MacDonald said...

@David Wong

You are using the wrong version. The 2.2.0 directory includes a Plugin that works from Cordova 2.2.0 and higher. If you are using 1.5.0 then try the 1.8.1 directory.

Unknown said...

Thanks Simon for your fast response. I got errors with TTS 1.8.1 and Cordova 1.5. I will try the other lineup of TTS and Cordova. I appreciate your help with the information.

Unknown said...

Hi Simon,

I tried Cordova 2.0.0 with TTS 2.0 as well as Cordova 2.2.0 with TTS 2.2. In both cases, window.plugins.tts.speak("The text to speech service is ready") produces no speech on my Android 2.2.1 smartphone. getLanguage() succeeds with a null string in the callback result. isLanguageAvailable() for "en_US" succeeds and a second call to getLanguage() succeeds with a null string in the callback result.

I am running the code as in your example. The window.plugins.tts.startup() succeeds with two invokations of the callback with result 1 and 2 (TTS.STARTED).

Do you have any thoughts on what might be wrong?

Thanks,
David

Unknown said...

Hi Simon,

I posted my last message a little too soon! The PhoneGap TTS plugin works after I installed Android's speech to text data (SpeechSynthesis Data Installer) from Google Play.

David

Unknown said...

Hi Simon,

I am using Cordova 2.2.0 with TTS 2.2. When I launch my app, my index.html calls window.plugins.tts.startup() after device is ready - I get TTS.STARTED - window.plugins.tts.speak() works. When I transition from my base index.html to a second help.html page, and transition back to index.html, I do the same window.plugins.tts.startup() after device is ready - now I do not get TTS.STARTED - fortunately window.plugins.tts.speak() continues to work.

When I transition from index.html to help.html, I added a call to window.plugins.tts.shutdown() which succeeds. Upon transition from help.html to index.html, I no longer get TTS.STARTED and window.plugins.tts.speak() no longer utters any sound.

Is there a need to call shutdown() in the way I've described above? Are resources being allocated but not released by not calling shutdown()? However, if I call shutdown() on transition from index.html to help.html, speak() no longer works after re-entry to index.html.

Thanks,
David

Simon MacDonald said...

@David Wong

Since you are going to continue using the TTS service when you get back to index.html I wouldn't bother calling shutdown.

Unknown said...

Thanks for your information Simon. The TTS plugin is working very well in a new app I am developing. Since I target my apps to Android as well as iOS, I wish for a similar Cordova plugin for the latter. Are you aware of any good options for iOS?

Unknown said...

Thanks for great plugin,This plugin works with SpeechSynthesis Data Installer from android market

Is there any way to do it without the use of SpeechSynthesis Data Installer..

Simon MacDonald said...

@rajan singh

Yeah, if your device does not come pre-installed with Google TTS then you will need to install a 3rd party TTS.

Simon MacDonald said...

@David Wong

Apple does not provide an SDK for TTS. You'd have to use a third party solution and wrap it in a plugin like:

http://www.speech.cs.cmu.edu/flite/

Unknown said...

Thanks for the lead, Simon. I'll investigate.

Unknown said...

For version 3.0 of cordova , the package name seems to have changed.
from
import org.apache.cordova.api.CallbackContext;
import org.apache.cordova.api.CordovaPlugin;
import org.apache.cordova.api.PluginResult;

to
import org.apache.cordova.*;

after making this change I am able to compile with cordova 3.0

Simon MacDonald said...

@Bhavesh Patel

Use this repo for 3.0.0 support.

https://github.com/macdonst/SpeechSynthesisPlugin

I will be doing a blog post on it soon.

Unknown said...

Hello ...

Is there someone that might have a link to a simple test app (in phonegap) that uses the above tts code ...I could download and test on my android Device ...

Thank you in advance ...

Michael T.

Simon MacDonald said...

@Michael

This plugin is very out of date. I've switched over to basing things off of the W3C Speech Synthesis spec. Here is a demo repo:

https://github.com/macdonst/ttsdemo

SANJAY CHAUDHARY'S BLOGS said...

How to get end of speak event.
I want to speak next string after end of currently playing one.
is there any event which give me end event?

Unknown said...

End of speak event for auto play next item after complete current one
this plugin can provide end event for current playing item is end to speak.