msmd lets you draw a rectangle over text on a macOS screen, recognizes
that region with Apple's built-in Vision framework, and emits the recognized
characters as keyboard events. An amber border remains around the capture region
while OCR and typing are active.
msmd-demo.mov
The first release targets macOS. Capture, OCR, and keyboard output are separate interfaces so Linux and Windows adapters can be added without changing the text tracking or command-line flow.
Use this only on sites and accounts where automation is permitted. Automated results are not meaningful typing scores and may violate a site's rules.
Install with uv
Install uv and make sure your Python build has Tk support. Then let uv create and synchronize the managed project environment:
There is no environment activation step. Run commands through uv run so uv
uses the locked project environment.
On first use, macOS will ask for two permissions for your terminal or Python:
- Screen & System Audio Recording, so the selected pixels can be captured.
- Accessibility, so keyboard events can be sent to the browser.
Enable them under System Settings → Privacy & Security, then quit and relaunch the terminal if macOS does not apply the permission immediately.
Run
Start with a dry run to verify OCR without typing:
Then run a single capture:
The interaction is:
- Drag a rectangle tightly around the typing-test text.
- Press Enter to confirm it, or Esc to cancel.
- During the three-second countdown, click the browser's typing area.
- Keep the target field focused while the recognized characters are emitted.
For a test whose viewport changes or scrolls, use experimental continuous mode:
uv run msmd --continuous --typing-delay 0.01
Stop continuous mode with Ctrl+C in the launching terminal. The tracker uses exact overlap between consecutive OCR frames and waits for two matching frames before accepting a completely replaced viewport. This reduces duplicate typing, but OCR mistakes or animated layouts can still produce incorrect input.
For a typing test with a fixed time limit, --duration implies continuous mode
and stops automatically after the requested number of seconds. The timer starts
after the focus countdown:
uv run msmd --duration 30 uv run msmd --duration 60 --typing-delay 0.01
Useful options:
--countdown SECONDS Time available to focus the browser
--typing-delay SECONDS Delay after each generated character
--scan-interval SECONDS Delay between continuous OCR frames
--duration SECONDS Stop continuous OCR after this many seconds
--language LANGUAGE Vision language, such as en-US or de-DE
Current limitations
- Region selection currently targets the primary display.
- OCR quality depends on contrast, font size, animation, and the selected bounds.
- Unicode events work for ordinary typing-test text, but a site that requires physical key codes may need a keyboard-layout-aware writer later.
- Continuous mode is intentionally conservative but cannot guarantee that noisy OCR will never retype text.
Development
Platform-neutral behavior lives in core.py; macOS-specific screenshot, Vision,
and keyboard implementations live in platforms/macos.py. A future Linux adapter
can use X11/Wayland capture plus Tesseract, while Windows can use Windows Graphics
Capture plus Windows OCR without changing the rest of the application.