Meta’s next AI might allow you to type without using your hands

Surface electromyography (sEMG) technology is advancing as a means of translating muscle activity at the wrist into digital commands, with potential applications ranging from augmented reality control to keyboardless typing.

Major breakthrough: Meta is releasing two groundbreaking datasets and benchmarks for sEMG-based typing and pose estimation as part of NeurIPS 2024, representing the largest open-source sEMG datasets ever compiled.

• The datasets include 716 hours of sEMG recordings from 301 consenting participants
• Each dataset contains 10 times more data than previous single-task, single-device collections
• State-of-the-art models for typing and pose estimation are being released alongside the datasets

Technical innovation: Surface electromyography measures electrical signals from muscle activity at the wrist, which machine learning models can interpret into digital commands.

• The wrist is particularly valuable for this technology due to the concentration of muscles controlling fine motor skills
• The technology has been demonstrated with Meta’s EMG wristband and Orion AR glasses prototype
• Users can perform actions like swiping, clicking, and scrolling while keeping their arm at rest

Dataset specifics: The two datasets, emg2qwerty and emg2pose, offer comprehensive data for advancing sEMG applications.

• emg2qwerty includes 346 hours of recording from 108 participants, capturing over 5.2 million keystrokes
• emg2pose contains 370 hours of data from 193 participants, including 80 million pose labels
• Both datasets support research into building generalized sEMG models that work immediately with new users

Performance metrics: The technology has achieved significant milestones in both typing and pose estimation capabilities.

• Character error rates below 10% have been achieved for typing when combining personalization and language modeling
• The vemg2pose model demonstrates only 1 cm error on new users for hand tracking
• Performance improves with about 30 minutes of individual user data for personalization

Future implications: These developments could fundamentally transform how humans interact with computing devices.

• The technology could enable typing without physical keyboards in any environment
• Hand pose tracking could work in conditions where visual tracking is limited, such as dim lighting
• The datasets may accelerate research in neuroscience and machine learning for motor control interfaces

Research opportunities: Meta’s release of these datasets opens new avenues for scientific exploration and technological advancement.

• The datasets enable research into complex generalization problems in sEMG
• Researchers can explore applications in domain adaptation and self-supervision
• The data could advance understanding of motor unit action potentials in neuroscience

Looking ahead: The continuing evolution of sEMG technology and its integration with machine learning could reshape human-computer interaction, though significant work remains to be done in improving accuracy, reducing latency, and ensuring reliable performance across diverse user populations.