Microsoft Research uses Kinect to translate sign language

Hearing impaired people around the world use sign language to communicate but there are lots of times where an interpreter is needed to translate sign language. Unfortunately, those human interpreters are rather limited in number. Today, Microsoft announced a new project from their Microsoft Research team in Asia that is trying to create a new and better way to translate sign language.

In a blog post, Microsoft states the team, working with another group at the Institute of Computing Technology at the Chinese Academy of Sciences, has created software that uses the Kinect for Windows hardware for its project. The Kinect hardware camera can track the movements of a person using sign language. The software then tries to "read" what the hand and finger movements are spelling. Microsoft adds:

The words are generated via hand tracking by the Kinect for Windows software and then normalized, and matching scores are computed to identify the most relevant candidates when a signed word is analyzed.

The software has two modes. The first, Translation Mode, is self explanatory. It translate the sign language from a person into either text or speech. The second, Communications Mode, has a hearing person using the software who types in words that are then displayed on screen as sign language via a 3D avatar. The person on the other end then uses sign language that is then translated to text or speech.

Microsoft says that while this software project only supports American sign language at the moment, it could be updated to include sign languages from other countries.

Source: Microsoft

Report a problem with article
Previous Story

Microsoft launches Outlook Web Apps for iPhone and iPad

Next Story

Microsoft defends against data sharing allegations

14 Comments

Commenting is disabled on this article.

Yeah but remember, they have to translate deaf people's name signs. Also, not every sign in the USA is the same. I remember seeing one sign for the color purple by a Michigan person and it's completely different from the standard sign for purple (shaking the P handshape).

Also, let's not forget the older deaf people who has stuck to older signs as well - like the current sign for 15 (similar to the "bring it on" sign with thumbs extended used by hearing people, but with one hand) .... the older sign for 15 was just "10" (shaking the "thumbs-up") and 5.

So, yeah maybe the Kinect needs a "learner" so the deaf person can train the Kinect to recognize the sign.

Kinect is being capable of doing so many innovative and cool things is just unbelievable. One piece of technology that has the ability to transform peoples live is so unique.

minster11 said,
That's pretty cool idea. It will be more impressive if Skype on Xbox will translate to the receiving end in text

I was trying to reply to your post. Sorry. Mobile messed up

Spicoli said,
Wouldn't using the keyboard be faster and more accurate?

Thought the same the last time the teacher asked me to write down something with a pen

This could be an amazing piece of kit if utilized properly. Would bridge equality of Deaf people in the workplace/business and allow deaf people to integrate even more. Im all for tech like this.