Archive for the ‘Application Development’ Category

On-Screen Keyboard Automation

Wednesday, December 4th, 2013

On-screen keyboards are such a pain in the butt. Fortunately, the Verizon FIOS remote app has a way type on the cable box’s on-screen keyboard through the iPhone’s keyboard. And Apple’s Remote app has a way to use the iPhone’s keyboard to type on the AppleTV’s on-screen keyboard. But unfortunately, the Remote app doesn’t seem to want to type when using XBMC on my jailbroken AppleTV. It is so frustrating using the on-screen keyboard to type in a search term! Right, right, right, right, down, select. Left, left, up, select. Right, right, right, down, down, select. NO! Wrong key! Right, right, right, down, down, select to delete. Left, left, left, up, select. What a pain! There has to be a better way! Introducing…. the On-Screen Keyboard plugin for Indigo!

Watch this video to see the On-Screen Keyboard plugin for Indigo in action. (coming soon)


The latest version of the plugin can be found here: OnScreenKeyboard.indigoPlugin

Version 1.0 – 12/4/2013 – Initial Release

Download the zip file and expand it on the Mac that you have Indigo running on. When you double-click the expanded plugin file, it should install into Indigo automatically. Navigate to Plugins -> On-Screen Keyboard -> Enable.

Set Up

A little pre-work is required for the On-Screen Keyboard, but if you already have Indigo controlling the device that you need to use an On-Screen Keyboard on, then you are probably already prepared to begin setting up your own Automated OSK.

In order to use an OSK, you need to use five basic functions on your remote: Up, Down, Left, Right, and Select. So, for the OSK Plugin, you need to have an Action Group configured for each one. As long as the Action Group “pushes the button”, the Plugin doesn’t care how the button actually gets pushed. Your Action Group could send a network message to the device, or it could do what mine does, and send an IR Blast to the device.

Once your Action Groups are configured, you are ready to create your OSK. Under Devices, click “New…” Choose Type: “On-Screen Keyboard”. Under Model, if you are creating an OSK for XBMC, simply choose “XBMC On-Screen Keyboard” – I have pre-configured the OSK for XBMC. If you need to control an OSK other than for XBMC, choose “On-Screen Keyboard”.

Note: To simplify the OSK setup for XBMC, I have hidden all of the details of the pre-configuration. For this guide, I will explain how to create an OSK for XBMC from scratch.

After choosing the Model: “On-Screen Keyboard”, the configuration dialog will appear. At the top of the config, you will be presented with your choices for how to navigate UP, DOWN, LEFT, RIGHT, and SELECT. For each choice, select the Action Group that is responsible for performing each of these actions.

The next section allows you to define the Variable that you wish to have the OSK “type”. You can choose a variable that you have previously created, or directly from the configuration dialog, you can create a new one (or even delete an existing one). I named my variable “xbmcKeyboard”.

The next part is the meat of the configuration. This is where you define the layout of the actual OSK. On your actual device, bring up the OSK. The keyboard layout that you are looking at first will be called the “–OSK1–” layout. This title is already pre-populated in your OSK definition (and can not be deleted or changed). Other keyboard layouts could include capital letters, special characters, number keys, and more.

The XBMC keyboard is eleven keys wide and five keys high. Each key needs to be separated by a space. I have defined the first row as:

DONE 0 1 2 3 4 5 6 7 8 9

The “DONE” key is a special key. We can call it anything we want, but it stands for the key to press when we are done typing on the keyboard. You will define this key later in the config.

The second row:

[--SHIFT/OSK1--] a b c d e f g h i j

The “[–SHIFT/OSK1–]” key definition is an instruction to change to the SHIFT keyboard layout, and return to the OSK1 keyboard layout after the very next keypress. (On XBMC, the first keyboard is all lowercase. Pressing SHIFT will display the uppercase keyboard. After choosing an uppercase letter, the keyboard returns to the lowercase keyboard.)

The third row:

[--CAPS--] k l m n o p q r s t

The “[–CAPS–]” key definition is an instruction to change to the CAPS keyboard layout (and stay there).

The fourth row:


The “[–SYMBOLS/OKS1–]” key definition is, again, an instruction to change to the SYMBOLS keyboard layout, and return to the OSK1 keyboard after the very next keypress. Now, the keyboard on the screen only shows one BACKSPACE key, but since the keyboard needs to be eleven keys wide, we need to define four BACKSPACE keys to keep the width the same.

The fifth row:


The “[–IPADDR–]” key definition would take us to the IPADDR keyboard. You’ll see later that we don’t bother defining this special keyboard, but we need to define the key anyway. “[SPACE]” is a reserved key name, and defines a space. Notice how there are many [SPACE] keys, as well as two [LEFT] and two [RIGHT] keys. Again, the multiple keys are defined to preserve our keyboard width.

This completes the primary keyboard layout. Now we need to define the other layouts for this keyboard, starting with the SHIFT/OSK1 keyboard.

DONE 0 1 2 3 4 5 6 7 8 9
[--OSK1--] A B C D E F G H I J
[--CAPS--] K L M N O P Q R S T

Pretty much the same as the OSK1 keyboard, but with uppercase letters.

DONE ) ! @ # $ % ^ & * (
[--SHIFT/OSK1--] [ ] { } - _ = + ; :
[--DONOTUSE--] ' " [COMMA] . < > / ? \ |

The SYMBOLS/OKS1 keyboard is a bit different, but not really. “[–DONOTUSE–]” is really just a placeholder that won’t match any keyboard input, meant to use as a spacer to keep the eleven by five keyboard layout intact. Notice how I used [COMMA] here for the comma. I could have typed in an actual comma (,), and OSK would have converted it to [COMMA] automatically.

I defined the “–IPADDR–” keyboard layout header, but I didn’t bother defining what that does. If I end up needing to type in an IP address, OSK will just use the numbers from the upper or lowercase keyboards, and the dots from the SYMBOLS/OKS1 keyboard.

That defines the keys on the OSK.

Next on the configuration dialog is to define the Starting Key and the Submit Key. For XMBC, this is the “DONE” key, so just type DONE in each field.

Vertical and Horizontal Multi-Keyspan Priority Modes take a little bit of research to figure out how to set. Let’s say I wanted to use the OSK to type “cats and dogs”. After navigating to the “s” key, I need to go three left and two down to get to the spacebar. I then need to get to the “a”, but how does the keyboard work to get there? If I push the UP button, do I go back up to the “z” key, which is the key I came down to the spacebar from? No, I go up to the “u” key, which is the left-most key above the spacebar. This is what the Vertical and Horizontal Multi-Keyspan Priority Modes are for. For XMBC, Vertical should be set to “Top”, and Horizontal should be set to “Left”.

Finally, there is the Spirit Board Mode checkbox. When the checkbox is checked, the OSK will be in “spirit mode”. While in this mode, the keys will be navigated to, but the SELECT button will never be pressed. So, what you will see will be the cursor navigate to each key that would be pressed, pause for a short period of time, and then move to the next key to press. You should make sure that your keyboard will work before turning off “spirit mode”. If you don’t, who knows what keys the OSK will press. That is, heaven forbid that when you submit your text to type on the screen that the cursor somehow navigates to something that you don’t want it to go to, and magically presses all of the keys in the right order to delete all your pictures, or buy a pay-per-view package, or whatever.

Putting it All Together

Now that your OSK has been defined, we need to set things up so you can make it type. Create a new Action Group. Choose Type -> On-Screen Keyboard Options -> Do On-Screen Keyboard. Device -> Choose the device you defined. Click OK. Running this Action Group will invoke the OSK Automation keypresses against the text in the variable that you have defined.

Assuming you have a Control Page, add a new control to display the variable value for the variable that you have defined. Client Action: Popup UI Controls. Do not put a Server Action here, as that will kick off the action before the variable has been set. Instead, create a separate button that will invoke the do-on screen keyboard Server Action.

It is also possible to use a Trigger that will fire the server action when the variable changes. I didn’t set it up that way for fear that I set the variable and then didn’t want the OSK to fire for whatever reason.

Future Enhancements/Wish List

If somebody really wants/needs any of these, I can be persuaded to work on them (I will absolutely work for food!). These are some ideas I came up with that I consider “nice to have”, but don’t really impact the effectiveness of the plugin for my use. Sure, some of them may make the act of “typing” more efficient, but do I really care about saving another two seconds when I use the plugin? Not really. Plus, it’s entertaining to watch the way it is now.

1. Might be nice/required to configure the sleep time for Spirit Mode. Other users may need to have a longer or shorter sleep time between key presses. Not sure. I guess I will find out if I get any feedback on this.

2. Somehow, compute the most optimal way to make use of the keyboard. For example, if I’m trying to type a word that starts with an upper-case letter and is followed by a lower-case one, it’s probably better to make use of the Shift Key. However, if the word I’m trying to type has two consecutive upper-case letters, maybe it’s more efficient to use the Caps Lock key.

3. Add “Keyboard wrapping” as an option. That is, if you went left on the left-most key on the keyboard, the cursor may jump to the right side of the keyboard. It might be less keystrokes to wrap around the keyboard than to travel all the way across it.

Using My Gifts to Make Life Easier

Wednesday, December 4th, 2013

God has blessed me with the gift of solving problems by means of software development. In my career as a programmer, I find most of my joy in writing applications that make peoples’ lives easier.

At home, I have never written anything for the purposes of making a living – it has always been because I had a need, or I envisioned a better way of doing something. For example, a very long time ago, I enjoyed playing darts. I hated doing the math for playing 301, so I wrote an app for the Palm Pilot called “Oh-One!” The app not only kept track of the score, but it also displayed a real-time chart that showed you the most optimal way to win the game. I released the game as “postcard-ware”, requesting people send me a postcard if they found the app useful. I got thousands of downloads, and about five postcards from around the world. My app appeared in some Palm Pilot Apps magazine in Japan, too, which was neat to see “David Garozzo” in the middle of a sea of Japanese characters.

With life happening, I rarely get the chance to complete my pet projects. At least, not to the point where it’s worth distributing. In the event that I do write something that I feel is worthy of making pubic, I will share the details with you. Hopefully somebody out there will be able to enjoy using something I have put together. If that ends up being you, please let me know about it!

Talking Titletrack

Sunday, July 9th, 2006

In September of 2001, I married my beautiful wife, Marie. When we married, I had about 400 audio CDs. Marie had about 150 CDs of her own. I had my CDs organized alphabetically in four LaserLine 90 CD holders, with some piled on the top. Even with so many CDs, it wasn’t too hard for me to find what I wanted. The worst part was when I purchased a new CD that started with the letter “D”, having to move all of the CDs to keep them in order.Image of CD cases with raised letter labels on them

My wife, Marie, isn’t able to manage her collection quite as easily as me – she’s blind. Her CDs were labeled with one inch raised letter tape on every jewel box. For her to find a CD that she wanted to listen to, she would have to try to keep her CDs as organized as possible, and then feel the cover of each jewel box to find the one she wanted to hear. As you can imagine, it wasn’t uncommon for her to find the wrong CD in a jewel box. Putting CDs away would require her to know which CD it was by listening to it (and identifying it) first.


After we got married, we had almost 550 CDs between the two of us. I had no interest in labeling all of my CDs for Marie – there had to be a better way.

I did some searching around, and I found this thing called a Slink-e. This niftly little box allows your computer to control multiple Sony CD changers. You can store all of your CDs in Sony CD changers, and your computer would take care of cataloging them, playing them, and more. How cool is that?
So I got to thinking. I’m a computer programmer. How can I take a system like this and make it easy for Marie to use.
I found two popular applications for the Slink-e – CDJ for PC, and TitleTrack Jukebox for Mac. Being a programmer, a Mac fan, having a couple of spare Mac’s in my house, and reading that TitleTrack Jukebox was AppleScriptable, my choice was simple. I decided to purchase a Slink-e with TTJ, three 200 CD changers, and start coding.
After not too long, I was able to hack together an application that I call “Talking TitleTrack.” It’s a RealBasic application that uses AppleScript to communicate with TitleTrack Jukebox and utilizes Apple’s SpeechLib Speech Synthesis Manager to speak all of the keystrokes and window information.
I’ve installed my little program on my old PowerCenter Pro 210. I put TitleTrack Jukebox in the Startup Items folder, along with a System Beep sound. I have no need for a monitor on the computer – I can use VNC or TimbukTu to control the computer if I need to.
When my wife starts up the computer, she waits for the System Beep. This lets her know the computer has finished starting up. Then she hits F14, which starts up my program, Talking TitleTrack.
The computer says “Getting Changer List…” When it’s done loading, it says “Search For”. Marie can then type what she wants to search for. It speaks each letter as each key is pressed, including “space”. And when she hits the delete key, it speaks the letter that was deleted, too.
Image of Talking TitleTrack Search For Music Window shows results for a search for 'Chapman'If she hits the “tab” key, the computer speaks “Search by Artist”. This is a list box, which can be changed to “Search by CD Title”.
Upon hitting the “enter” or “return” key from either the “search for” or “search by” prompt will kick off the search. The computer will sound “Searching ‘by Artist’ for ‘Chapman’.”
The application will then return the list of results, with the computer speaking “Search results: ‘Declaration, by Steven Curtis Chapman’.” The results are displayed in a list box. By using the up and down arrow keys, she can hear each found selection. When she hits “enter” or “return”, TitleTrack Jukebox will clear its current playlist, add the selected CD, and begin playing the CD.
As you can imagine, this application has been invaluable for my wife, giving her some freedom to quickly and easily access all of our CDs. It’s been awesome!
In addition to being able to play any CD at any time, I created some static playlists for Marie. I assigned the playlist files to a function key. When she hits that key, the playlist will load straight into TitleTrack Jukebox. This has nothing to do with my program, but what I did do was to add a “What’s Playing” feature. When Marie hits Command-I, Talking TitleTrack will announce “Now Playing: ‘Let Us Pray’ by ‘Steven Curtis Chapman’ on ‘Signs of Life’.”

If Marie is curious to see what selections are coming up in her playlist, she can hit Command-P. This will bring up a window that will read the current Playlist. Using the up and down arrow keys, Marie can hear each selection. It doesn’t allow her to pick the song she wants to hear; only see what’s coming up.
Image of VCR-Record a ShowFinally, I’ve added one more really cool feature for Marie. Since Marie is a huge fan of the Thursday night lineup on TV, as well as all of the Law and Order shows, she’s always asking me to set the VCR to record this or that on TV. Unfortunately, we don’t have a talking VCR. (we do have a talking microwave, though) One of the neat things that the Slink-e does is to give the computer the ability to send IR commands out to control various devices like a remote control. Using TitleTrack Jukebox, I configured a remote control for our PROSCAN VCR. Then, I created a dialog box in Talking TitleTrack that Marie can bring up by hitting Command-R. This will announce “Program VCR: How Often: One Time”. Using the keyboard, Marie can fill out all of the information required to record her show. After all of the information has been entered in, Talking TitleTrack, via TitleTrack Jukebox, will send IR commands to program the VCR, just like I would enter it in using the remote control! I don’t know if Marie ever thought she’d be able to be able to program the VCR on her own again.
There are probably plenty of enhancements that I can make to Talking TitleTrack, but as it stands, it functions very well for Marie. I would imagine that it would also work great for anybody who wanted to eliminate a monitor. Of course, then you’d lose all of the coolness that the TitleTrack Jukebox brings you, but that’s your call.
When TitleTrack Jukebox 3.0 was released for OS X (the upgrade is fantastic!), I made the appropriate changes to Talking TitleTrack to conform to the new message responses. I left the code smart enough to work with either version, and compiled it as a Carbon app, so Talking TitleTrack can be used with any version of TitleTrack Jukebox, on OS 9 or OS X.
I would like to thank Dick Dinkelspiel, the author of TitleTrack, Inc. His support and dedication of his product has been absolutely incredible. Without his assistance and encouragement this project wouldn’t be here today.
If you would be interested in trying out Talking TitleTrack, I have it here for you to try. Launch TitleTrack Jukebox first, and then run Talking TitleTrack. Remember – this application is meant for a blind person. Don’t expect to see anything pretty; just plain and simple dialog boxes. Try turning off your monitor or closing your eyes for the full effect.
If you find value from Talking TitleTrack, please send a small donation my way by using the PayPal link on this page. I’d love to take Marie out to dinner (pizza delivery would be fine, too), or buy a new CD for us to listen to.
Be forewarned that I don’t expect the VCR part to work for you. It’s configured to run for the PROSCAN VCR, which you probably don’t have. Plus, it takes some setup in TitleTrack Jukebox for it to learn your remote control. You’ll have to read the TitleTrack docs to figure all that out. However, if you would be interested in getting something like this working with your VCR, or maybe you have some other ideas, PayPal me a donation, and I’ll be happy to work with you to get things set up for you.
I hope somebody else finds as much value from Talking TitleTrack as my wife Marie and I have.
Click here to download Talking TitleTrack.

Speech Recognition Mod

Sunday, July 9th, 2006

Speech recognition on the Mac is something that I have toyed with on and off since it was first built-in to the Mac OS back in 1993. Having the computer recognize my voice definately has its “cool factor”, but sometimes it’s just easier to use the mouse.

The other day, I was watching Extreme Makeover: Home Edition. The episode was particularly interesting to me, because the makeover was being done for a man who lost his sight. As usual, they did some really amazing things for this family’s home. One thing that got me curious was the system that they installed that allowed the man to say something like “Computer, what temperature is the house?” and “Set the house temperature to 75 degrees.” and have the computer respond and act on his voice.


At the end of the episode, they said you could go to their website and learn all about the products used in the show. I immediately checked it out, and learned that the system that they used was HAL by Home Automated Living.


After doing some research on HAL, I learned that this system was little more than a voice recognition front end for an X10 home automation system. Sure, it had some other features, but that was basically the foundation of it. Being an Apple guy, I thought “big deal”. I could do that with AppleScript, the built-in Speech Recognition software, and one of the Mac X10 software apps.


I guess now is a good time to mention that my wife, Marie, lost her sight back in 1997 to Optic Neuritis. She was diagnosed with MS in 1998. I have a strong background in computer programming, and I enjoy using my gifts to write applications that can make her life easier. For example, read about how I set things up to allow her to have access to our huge CD collection with Talking TitleTrack.


Getting excited again about voice recognition and how I can enable my wife to be able to do more on her own, I decided to write an AppleScript that will go out and get the weather, and then read it aloud. I dropped it in the Speakable Items folder, and now I can say, “Computer, weather report.” Very cool.


Excited about my new script, I sat Marie in front of the computer, and had her say “Computer, weather report.” Nothing happened. She said it over and over, and nothing happened. I had her try one of the default commands “Computer, what time is is?” Nothing. She tried over and over, getting more and more frustrated, and couldn’t get it to work. Of course, it would pick my commands up immediately, which annoyed Marie even more.


Now I remember why I didn’t write anything like this before. I tried using it with Marie a few years ago, and ran into the same problem. My Mac just doesn’t want to listen to Marie’s voice.


Instead of giving up on the whole idea, I tried to figure out why Marie’s voice couldn’t be recognized by my Mac. My wife’s voice is very young sounding. I searched the web, and found some information that lead me to believe that higher pitched voices aren’t as easily understood by speech recognition applications. So, I asked Marie to try and talk in a deeper voice. After a few tries, the computer responded to her command! Very exciting!


Unfortunately, it is very difficult (and annoying) for Marie to speak in a lower voice every time she would want to tell the computer to do something. The computer didn’t pick it up every time, and she just sounds and feels silly talking that way. So I thought – I wonder if I can have Marie speak in her normal voice, and have the computer lower her voice programatically before passing the audio into the Speech Recognition application? Then Marie could speak in her normal voice, but the computer would “hear” a lower pitched voice that it could hopefully understand.


At this point, I turned to the Mac audio gurus – Rogue Amoeba. These guys are amazing. Not only do they put out solid applications, but they’re very reasonably priced, and they have top notch support.


I posted a message on their forums describing my problem, and they quickly introduced me to Audio Hijack Pro. With this program, I can hijack audio from any input or even any application, apply any set of filters I want to it, and then make that altered audio available as a new audio channel (called Soundflower). This is exactly what I was looking for!


I downloaded the demo version of AHP, and set up a test. The first thing I did was to see if the Speech Recognition application would be able to work through the new audio channel that AHP created for me. That seemed to work just fine. So far so good. Then I needed to apply a filter to pitch the sound down. At first I tried Apples new AUPitch control. Unfortunately, that thing’s a beast, and brought my Pismo down to a crawl. There’s no way my G3/500 is going to handle that kind of work.


I posted my results to Rogue Amoeba’s forums, and they pointed me to an older plug-in called MadShifta. It’s a freeware plugin written by Tobybear Productions and SmartElectronix.


I downloaded and installed the plug-in. I configured Audio Hijack Pro to use the MadShifta plug-in and set the pitch control. Much better performance with this plug-in compared to the AUPitch control. I sat my wife in front of my computer, and crossed my fingers.


“Computer, what time is it?” Worked on the first try!


“Computer, weather report?” Worked on the first try!


The hack/mod worked perfectly!


Knowing that Marie will now be able to use the Speech Recognition application, I am very excited about writing more Speakable Itemsfor her, and eventually hooking an X10 component up here and there, too.


A very speical thank you goes out to Rogue Amoeba for your guidance. Thanks to Tobybear Productions and SmartElectronix, too, for your free MadShifta plug-in.


Here are the details for this hack/mod:


  • Install Audio Hijack Pro from Rogue Amoeba
  • Install MadShifta into ~/Library/Audio/Plug-Ins/VST
  • From Audio Hijack Pro, choose Install Extras, and install Soundflower
  • From Audio Hijack Pro, create a new Session
  • Set that session’s Audio Source to “Audio Device”
  • Set the Input Device to your microphone
  • Set the Output Device to “Soundflower (2ch)”
  • Click on Effects
  • Insert the VST Effect -> MadShifta
  • In the MadShifta settings, adjust the Tune to -2. Your settings may vary.
  • Launch System Preferences -> Speech
  • Enable Speakable Items
  • Set the Microphone to “Soundflower (2ch)”
  • Adjust other settings in this dialog as appropriate
  • Speech Recogntition should now “hear” the voice shifted in real-time by the MadShifta plug-in through Audio Hijack Pro!