# The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend - David vonThenen

## Executive summary

This talk details acoustic keystroke logging—a method of intercepting typed information purely from sound rather than physical interception. The speaker outlines how deep learning models can be trained using spectrographic images derived from recorded key presses to classify specific letters (e.g., 'S'). While demonstrating the high accuracy of single-keyboard attacks (100%), the presentation shows that multi-keyboard logging is challenging but feasible, especially when combined with context prediction and spell-checking algorithms. The session concludes by emphasizing defensive measures, including two-factor authentication using physical keys and implementing strong, unique, offline password policies.

## Key takeaways

- Acoustic Keystroke Logging Mechanism: The attack relies on machine learning audio classification. Audio files (linear 16 format) are converted into spectrographic images (frequency over time, visualized as a heatmap), which serve as the input for training models like Convolutional Neural Networks (CNNs).
- Multi-Keyboard Attack Complexity: While single-keyboard classification can achieve 100% accuracy, using multiple keyboards significantly lowers confidence scores. The problem is made solvable by decomposing the text based on space delimiters and employing spell-checking/context prediction (e.g., predicting 'hello people' from partial sound inputs).
- Defensive Strategies: Defense requires layered security: use physical two-factor authentication keys (like YubiKey) instead of SMS; utilize unique, complex passwords that are not known to the user; and be aware of potential signal interference or noise.

## Technical details

- Data Preparation for ML: The process involves taking raw wave files (linear 16 format, e.g., 44.1 kHz) and generating spectrographic images. The data must be split into training, validation, and test sets (typically 80/10/10). For robustness, the model training incorporates noise addition and time shifting variability.
- Model Architecture and Training: The classification uses a deep learning approach (PyTorch) on spectrographic images. The speaker notes that while training requires significant resources (e.g., H100, ~12 hours), pre-trained models in safe tensor format are available for running inference on standard laptops.
- Contextual Attack Improvement: To improve accuracy beyond simple classification, the system can be designed to use spell-checking services (e.g., `spellch check`) on all possible combinations of predicted characters within a word segment, drastically reducing the search space and increasing the probability of identifying the intended message.

## Practical implications

- Implement physical two-factor authentication (e.g., hardware keys) to mitigate risks associated with remote acoustic interception.
- Adopt a policy of using unique, complex passwords that are generated and stored offline, preventing social engineering attacks.
- Be aware of potential signal vulnerabilities in IoT devices or wireless networks, as these can be exploited for data capture.

## Topics

Security, Machine Learning, Acoustic Signal Processing, Cryptography, System Hardening, GitHub Repo, Original Research Paper

Source: https://www.youtube.com/watch?v=23No-rHp0B8