TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Q&A: Pushing the Next Frontier in Handwriting Recognition

MyScript uses machine learning to build highly accurate handwriting recognition algorithms. The company says its technology works with 95 percent of written languages as well as mathematical equations and music.

• By Linda L. Briggs
• September 19, 2016

MyScript uses machine learning to develop technology that can accurately recognize handwriting -- not only in English, the company says, but in 95 percent of written languages. Upside recently interviewed Gary Baum, a VP at MyScript, about the next frontiers in handwriting technology.

Upside: Tell us a little bit about MyScript and its genesis.

Gary Baum: MyScript develops highly accurate handwriting recognition and digital ink management technology. Our mission is to enable digital writing using advanced, multimodal human-machine interfaces.

MyScript Labs was founded by our CTO, Pierre Michel Lallican, almost 20 years ago. Its sole focus was enabling a human to write on a device and have that input turned into useful digital information -- digital writing. Artificial intelligence techniques that mimic the human brain were developed and trained to "read" handwriting.

Over time, we created recognition for more languages, expert system refinements, and other additions to what the software can recognize. We are making handwriting input interactive and easy to use.

What is "digital ink management" technology?

We developed a way to interact with and manipulate digital ink that provides a dramatic improvement in the human interface with a device. We call this technology Interactive Ink; it includes real-time predictive handwriting recognition, editing in ink or digital mode, and gesture control -- all driven by neural network architectures.

Interactive Ink creates an intuitive and natural human-machine interface for writing not only text but mathematical equations, hand-drawn graphical structures, and musical notes across many productivity platforms. The natural user interface can dramatically increase creativity and efficiency.

Users can write block or cursive input on their devices in 95 percent of the world's languages, and it will be instantly interpreted into digital text. Advanced mathematical equations can be handwritten -- including symbols -- and immediately recognized or solved. Charts, maps, and diagrams can be drawn by hand and turned into digital shapes that are instantly digitized into an editable, searchable, and storable form.

How are handwriting recognition and artificial intelligence converging?

Neural networks are often applied to challenging problems that cannot be solved with traditional algorithms, and handwriting recognition falls into this category -- it is enabled through the use of artificial intelligence techniques. For almost 20 years, MyScript Labs has been perfecting and refining this technology as it applies to the complexities of handwriting recognition.

The way the human brain works when reading inspires our system, which depends on a large quantity of input parameters. Those parameters can be estimated using learning techniques based on millions of real samples.

Our process isn't limited to written text; it also applies to graphics, diagrams that can include shapes and text, math equations, and musical notation. As productivity gains are realized, more natural and intuitive applications can be developed, bringing more value to mobile devices.

How is your technology available, and how is it being used?

It's available in MyScript's developer portal and showcase applications, including MyScript Calculator. Developers have optimized and used our technology in a variety of applications.

For example, minimizing driver distraction is a key concern of the automobile industry. MyScript technology allows the user to write one character on top of another -- including cursive characters -- so that words and sentences can be written while the driver's eyes stay on the road. The letters are superimposed on top of each other. This award-winning "superimpose technology" has been well received by the industry.

IoT applications such as watches and remote controls also benefit from this technology. Applications include note-taking, math equation recognition for education applications, and musical notation.

How are analytics applied to handwritten text in the process of digitizing it?

MyScript recognition technology mimics how the human brain works when reading.

The complex task of reading can be divided into a number of processes -- segmenting a word into its constituent characters, recognizing the characters, and resolving ambiguities based upon contextual and linguistic information. The reading process relies on alphabetic knowledge, phonetic information (how a word sounds), lexical information (how a word is actually used), and grammatical or semantic knowledge, among other things.

Our most recent innovation is Interactive Ink. It provides a dramatic improvement in how the user interacts with the device and with digital ink. Now the digital ink itself takes on the meaning semantics, so that the input can be edited, moved, or reflowed in a manner we expect. Digital ink can provide content creation capabilities. Often, it's far easier to draw complex diagrams or write math equations by hand rather than other input methods.

Voice recognition is getting better, so why is the time ripe for handwriting recognition technology?

The model with electronic devices used to be that the user adapted to the device. That's no longer true. Users now expect their devices to cater to their needs and use patterns. In fact, the human-machine interface often dictates the success of a device or application.

Users have also shown an insatiable desire for mobility with their technical devices, giving rise to the amazing growth of the tablet and smartphone markets. These smaller devices were first used mostly for media consumption. As they move rapidly into the workforce, where productivity is the focus and voice use isn't always appropriate, other forms of content creation become necessary. That, in turn, means a need for better handwriting recognition.

What are some examples in which handwriting recognition is the most appropriate input method?

Digital handwriting is becoming a prominent way to input content in situations where using a keyboard is difficult and voice recognition isn't feasible -- perhaps the user is in a cubicle among many others or on a noisy subway. There are many instances where the pen or finger is required or desired -- or is just more intuitive or natural.

What are the next frontiers in this area?

We are very excited about Interactive Ink technology. It's the result of many years of work, including user interface research. It's the underlying technology behind MyScript Calculator, extended not only to math processes for calculation but also to text input for note-taking and graphics input for diagramming and document creation. Interactive Ink adds semantics to digital ink that can then be edited with immediate feedback.

Speaking of machine learning, a recent article in Wired magazine posited that in the future, we will train computers via machine learning. How does MyScript's technology tie into that prediction?

Today the most interesting and beneficial technical solutions are built on neural network techniques. There are many input parameters that are difficult to process independently using traditional algorithms. We can achieve a significant benefit by processing them using machine learning -- or more accurately, neural networks.

It's the learning process itself that is so intriguing. MyScript recognition technology is based upon millions of real-world samples used to train the expert systems.