How it works
From webcam to spoken words in about a second. The steps play on their own; click one to stop and read it.
{{stepTitle}}
{{stepBody}}
ASL is its own language
Sign order follows ASL grammar, not English. Signify shows both, so signers can trust what's being said for them.
Frontend ↔ model contract
The local app streams this to the page about 30 times a second over a WebSocket, alongside the camera frames.
{ "type": "state", "mode": "auto", "kind": "word", "label": "PAIN",
"stable": true, "word": { "label": "PAIN", "confidence": 0.91 },
"letter": { ... }, "text": "MY HEAD PAIN", "paused": false }
Field names from the bridge in Blake_frontend/server/ws_bridge.py.
Architecture
Under the hood, live
The raw engineering view of the same webcam feed: the 21 hand landmarks MediaPipe tracks, the skeleton image the letter model classifies, both models' scores against their confidence bars, the router's decision, and every message the backend sends. It needs the Signify bridge running on this laptop.
Top-1 accuracy on 250 signs (88.1% top-5)
Test set: 14,396 clips from 3 signers held out of training. Letters A–Z: 93.7%Sign-to-caption time
Letters commit in ~0.4 s, words in ~1 s (estimated from the confidence settings)Training clips used
From Google’s asl-signs dataset (Kaggle Isolated Sign Language Recognition)Built with
The build story
{{q.body}}
Privacy
Video never leaves the laptop, and nothing is recorded or stored. Only text you choose to have spoken is sent to ElevenLabs to make the voice. One tap pauses the camera for privacy.