The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend - David vonThenen
Summary
This talk details acoustic keystroke logging—a method of intercepting typed information purely from sound rather than physical interception. The speaker outlines how deep learning models can be trained using spectrographic images derived from recorded key presses to classify specific letters (e.g., 'S'). While demonstrating the high accuracy of single-keyboard attacks (100%), the presentation shows that multi-keyboard logging is challenging but feasible, especially when combined with context prediction and spell-checking algorithms. The session concludes by emphasizing defensive measures, including two-factor authentication using physical keys and implementing strong, unique, offline password policies.
Key takeaways
-
Acoustic Keystroke Logging Mechanism
16:22
The attack relies on machine learning audio classification. Audio files (linear 16 format) are converted into spectrographic images (frequency over time, visualized as a heatmap), which serve as the input for training models like Convolutional Neural Networks (CNNs).
-
Multi-Keyboard Attack Complexity
28:12
While single-keyboard classification can achieve 100% accuracy, using multiple keyboards significantly lowers confidence scores. The problem is made solvable by decomposing the text based on space delimiters and employing spell-checking/context prediction (e.g., predicting 'hello people' from partial sound inputs).
-
Defensive Strategies
40:50
Defense requires layered security: use physical two-factor authentication keys (like YubiKey) instead of SMS; utilize unique, complex passwords that are not known to the user; and be aware of potential signal interference or noise.
Technical details
-
Data Preparation for ML
982s
The process involves taking raw wave files (linear 16 format, e.g., 44.1 kHz) and generating spectrographic images. The data must be split into training, validation, and test sets (typically 80/10/10). For robustness, the model training incorporates noise addition and time shifting variability.
-
Model Architecture and Training
1154s
The classification uses a deep learning approach (PyTorch) on spectrographic images. The speaker notes that while training requires significant resources (e.g., H100, ~12 hours), pre-trained models in safe tensor format are available for running inference on standard laptops.
-
Contextual Attack Improvement
1780s
To improve accuracy beyond simple classification, the system can be designed to use spell-checking services (e.g., `spellch check`) on all possible combinations of predicted characters within a word segment, drastically reducing the search space and increasing the probability of identifying the intended message.
Mentioned resources
- GitHub Repo
- Original Research Paper
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.