2026/ Accessibility · Computer Vision/ Amazon Industry Program

3ayn

An AI assistive app for blind and low-vision users. I own the social-intent-detection module, the part that notices when someone is waving at you, approaching you, or trying to get your attention, and says so out loud in Arabic before you have to ask.

Try the demo View on GitHub ↗

Interactive demo

Simulated camera feed · pick a voice command

3ayn · social intent detection

Ready · awaiting voice command.

// narration output appears here

At a glance

My role: Social-intent detection module
Stack: MediaPipe · PyTorch · Java
Team: Amazbytes · Amazon Industry Program
The gap: Proactive, not point-and-ask

How the module works

Camera to spoken sentence, on-device

A wave only exists across time, so the classifier reads a window of landmarks, never a single frame.

Code

The two things that caused every early false alert

social_intent/landmarks.py partial bodies must not become gestures


MIN_VISIBILITY = 0.6
REQUIRED = ["left_shoulder", "right_shoulder", "left_wrist", "right_wrist"]

def usable(pose):
    # MediaPipe still returns landmarks for a body half out of frame,
    # with low visibility scores. Trusting those was the #1 false-wave source.
    return all(pose[j].visibility >= MIN_VISIBILITY for j in REQUIRED)

social_intent/intent_rules.py is the attention aimed at me?


def directed_at_user(track, window):
    # approach = bounding box growing steadily across the window
    growth = track.box_area[-1] / max(track.box_area[-window], 1e-6)
    approaching = growth > 1.25

    # a wave across the room is only social if they are facing you
    facing = abs(track.shoulder_yaw) < 35        # degrees off-axis

    return approaching and facing

Someone waving at a person behind you is the case that makes a naive gesture classifier feel broken.

Inside the repo

Structure · the parts I wrote are marked

▸app/Android client (Java)
□MainActivity.javacamera + audio lifecycle
□VoiceRouter.javamaps the 5 commands to modes
□NarrationQueue.javaTTS priority + interruption
▸social_intent/my module
□landmarks.pyMediaPipe pose + hand extraction
□gesture_model.pyPyTorch temporal classifier
□intent_rules.pyapproach / wave / attention logic
□train.pytraining loop + augmentation
□export_tflite.pyon-device conversion
▸narration/Arabic-first phrasing
□templates_ar.jsonArabic narration templates
□templates_en.jsonEnglish fallback
□requirements.txtmediapipe, torch, numpy
□README.mdsetup + demo instructions

Skills, in context

Where each one actually showed up

MediaPipe Landmark extraction, plus the visibility gate in landmarks.py above.

PyTorch Temporal classifier over landmark sequences, exported to run on the phone.

Computer Vision Box-growth as an approach signal; shoulder yaw to test whether attention is aimed at the user.

Java Android camera lifecycle and the TTS interrupt queue, which mattered more in testing than model accuracy.

Next project AI-Driven SOC Pipeline