Telesense
Multimodal Behavioral Analysis for Telehealth
A second observer for recorded sessions — timestamped, auditable, and explicit about what it cannot tell you.
- 3Modalities fused
- Per-frameConcern localization
- Local-firstRecordings stay on device
Over video, clinicians lose many of the nonverbal cues they rely on — facial expression, tone of voice, pauses, how a patient's affect shifts mid-conversation. Telesense takes a recorded session and returns timestamped flags for moments that may need follow-up, alongside an emotion timeline, protective factors the patient mentioned, and intake topics that were never covered.
What it does
- Upload a session and get timestamped flags for moments that may warrant follow-up, scored across audio, video and transcript.
- A synchronized review workspace: video playback, audio waveform, searchable transcript, and per-modality signal timelines that seek to the moment.
- Whisper transcription with model-derived word timestamps, so every passage links back to a point in the recording.
- Protective factors the patient mentioned and intake topics never covered, from transparent phrase-matching rules that send nothing to an external service.
- An emotion timeline and combined flag counts, with the engine that produced them labelled in the interface.
- A trainable multimodal model in the repository: per-modality encoders, cross-modal attention fusion, and a TCN + Transformer temporal encoder scoring every frame.
- A separate unsupervised track — temporal VAE plus contrastive learning — that surfaces unusual moments without any labels.
- JSON export, a sample library open without a key, and a gated upload path against a Fly.io CPU backend.
How it works
A session is uploaded and stored locally. The backend extracts audio with ffmpeg, transcribes it with Whisper, then runs one of two engines. The deployed demo runs the pretrained pipeline: Wav2Vec2 (SUPERB) scores three-second windows of speech for vocal emotion, RoBERTa (GoEmotions) scores transcript passages across 28 labels, and a keyword detector plus temporal smoothing turn those scores into timestamped flags.
The second engine is the Telesense model itself, which runs when a trained checkpoint is configured. Each modality goes through its own pretrained encoder — Wav2Vec2 for audio, ResNet-50 or DenseNet-121 per frame for video, DistilBERT for text. The features are projected to a shared size and passed through a multi-scale temporal encoder that pairs a TCN with a Transformer, so short patterns like a pause or a shift in tone and longer arcs across the session are both in reach. A localization head then scores every frame, and high-scoring frames merge into segments.
Localizing in time rather than emitting one label per session is the design decision the rest follows from: a reviewer can jump to the moment and judge it, and frame-level scores are auditable in a way a single verdict is not. Protective factors and missing intake questions stay deliberately rule-based — phrase matching is predictable and inspectable. The backend binds to 127.0.0.1 and keeps uploads, extracted audio and results on the local filesystem.
Python, PyTorch, FastAPI and a React + Vite frontend, over Whisper, Wav2Vec2, RoBERTa, DistilBERT and ResNet/DenseNet encoders. Docker for the backend image, Vercel for the demo frontend, Fly.io for the gated CPU backend.
- Python
- PyTorch
- FastAPI
- React
- Docker
- Vercel
A note on the numbersTelesense is a research prototype. It does not diagnose, screen for, or assess risk of any condition, and its output — including flags, protective factors and missing-question prompts — must not be used to make clinical decisions. The public demo runs an uncalibrated pretrained fallback on CPU, not a trained Telesense checkpoint and not a diagnostic classifier; it flags readily and produces both false positives and false negatives. Severity labels are uncalibrated application thresholds, not clinical cut-offs, and a model score is not the probability that someone has a condition. Anyone at risk needs evaluation by a qualified clinician; in an emergency, call local emergency services.
Have a matchup in mind?
Open the demo