Egocentric video annotation

Egocentric Video Annotation Services

Egocentric video annotation labels first-person footage from head-mounted or wearable cameras: the wearer's hands, the objects they touch and the actions they perform. VIPRO tags hand-object interaction, 21-point hand keypoints and Ego4D-style action phases, with temporal boundaries audited against a tIoU target of 0.75 or higher.

First-person footage from head-mounted and wearable cameras, labelled for hands, active objects and actions, so robots and assistants learn from the wearer's point of view.

Start a Pilot Explore Services

Compatible workflows

Tools we can work with

Roboflow
SuperAnnotate
Darkhorse
Roboflow
SuperAnnotate
Darkhorse

TL;DR

First-person (wearable camera) footage needs its own rules for hands, objects and head motion. Image or exocentric video guidelines don't transfer.
We label hand–object interaction, 21 keypoints per hand with visibility flags, and Ego4D-style actions with pre-action, point-of-no-return and post-action phases.
Temporal boundaries are audited against a tIoU ≥ 0.75 quality target, measured per batch. It's a target, not a guarantee.
Delivery in Ego4D Temporal Action JSON, COCO JSON or CVAT XML, proven on our 1M+ frame factory egocentric project.

First-person data

What egocentric data is, and how it differs from exocentric video

Egocentric video is recorded from a camera worn by the person doing the task, so the footage shows the world from their point of view. Exocentric video watches the person from outside. The same action looks very different from each viewpoint, and the labelling rules have to change with it.

Egocentric versus exocentric at a glance

Aspect	Egocentric	Exocentric
Camera position	Worn by the person: head, chest or glasses	Fixed or handheld, watching the scene from outside
What dominates the frame	The wearer's own hands and the objects they touch	Whole bodies, people and the wider scene
Motion	Constant head motion, fast pans, motion blur	Usually steadier; the subject moves, not the camera
Typical occlusion	Hands covering the object being used	People and objects blocking each other
Main labelling question	What is the wearer doing, with what, and when?	Who is where, and what are they doing?

Public egocentric datasets

These public benchmarks shaped how the field labels first-person video. We can deliver labels in schemas that follow their conventions.

Dataset	Generally known for
Ego4D	A large, multi-country collection of daily-life first-person video with benchmarks for episodic memory, hand–object interaction and forecasting.
Ego-Exo4D	Skilled activities recorded at the same time from a wearer's camera and external cameras, for linking first-person and third-person views.
EPIC-KITCHENS	Unscripted kitchen activity filmed with head-mounted cameras, widely used for verb–noun action recognition and anticipation.

Hands and objects

Hand–object interaction labelling

In first-person footage the hands are the subject. Most of the useful signal sits where a hand meets an object: what is touched, by which hand, and how.

Contact states

Each hand gets a contact state on every labelled frame: no contact, touching a portable object, touching a fixed object, or touching another person. The object in contact is linked to the hand by ID, so a model can learn which hand is doing what, rather than just seeing that a hand and a wrench share a frame.

The guideline defines contact visually. It starts on the first frame where the hand and object visibly meet, and doesn't count a hand that is merely near the object. Tool-mediated contact, such as a screwdriver touching a bolt, is labelled as its own relation when your model needs it.

Active objects

The active object is the one being used, not every object in view. We box or segment active objects and keep their IDs stable across the clip, so the object picked up in one shot is the same object set down several seconds later.

Hand pose

Hand keypoints: 21 per hand, with visibility flags

We use the common 21-point hand layout: the wrist plus four joints on each finger, from the base knuckle to the fingertip. Left and right hands are labelled as separate instances.

Visibility flags

Every keypoint carries a visibility value: visible, occluded but inferable, or not labelled. Fingers curled around a tool are usually occluded but inferable. The guideline says when an annotator may place an inferred point and when the point must stay unlabelled, so the model never trains on guesses presented as facts.

Handedness and identity

Hands are tagged as the wearer's left or right, or as belonging to someone else in view. Handedness is checked on playback because a mirrored or partial view can easily fool a single-frame judgement.

Temporal actions

Ego4D-style action tagging with three phases

A single start and end time hides the structure models increasingly need for anticipation and robot learning. We can split each action into three phases.

Pre-action, point of no return, post-action

The pre-action phase covers the approach and preparation: the hand reaching toward the object. The point of no return is the frame where the action is committed and its effect begins, such as the moment a cap starts turning. The post-action phase covers release and the visible result. Each phase boundary is a frame index tied to a written visual cue.

Phase labels are optional. Recognition-only projects may need just start and end, while anticipation and manipulation models benefit from the full structure. We decide this during calibration.

Verb, noun and object schemas

Actions are written as verb–noun pairs from a closed vocabulary, for example pick_up + torque_wrench. Verbs are defined by motion and effect, not by intent. Nouns map to the object classes used in the spatial labels, so the temporal and spatial layers share one taxonomy. New verbs or nouns go through a change request instead of being invented mid-batch.

For a plain-language introduction, read egocentric video annotation explained.

Boundaries

Narration and action boundary rules

Free-text narration and structured action labels serve different purposes, and both need rules to be consistent.

Narration

When narration is in scope, annotators write short present-tense sentences about the wearer's activity, timestamped at the moment the event happens. A style guide fixes person, tense, vocabulary and level of detail, so narrations from different annotators can be compared and parsed.

Boundary rules

Rules cover the minimum action length, the gap that separates two instances of the same action, how to label simultaneous actions by the left and right hands, and what to do when an action is interrupted and resumed. Ambiguous edges are flagged rather than forced.

Hard footage

Handling motion blur and hand occlusion

Head-mounted cameras swing fast, and hands constantly cover the objects they hold. These frames are where egocentric datasets are won or lost.

Motion blur

Frames blurred by head motion get a blur attribute. Geometry is drawn to the visible extent of the object, not the smeared trail. When blur makes an object unreadable, annotators follow the track from neighbouring sharp frames and mark the frame as low confidence instead of skipping it or guessing a class.

Hand occlusion

An object hidden inside a grasp keeps its ID and is marked occluded. Its box covers the visible part unless you've asked for amodal labels. When an object leaves the view and returns, re-identification is checked by a reviewer before the original ID is reused.

General tracking and occlusion rules for third-person footage are on our video annotation page.

Quality control

QA for egocentric work: temporal IoU and playback review

Spatial labels are reviewed frame by frame. Temporal labels are reviewed on playback and measured with temporal IoU (tIoU), the overlap between delivered and reference action segments divided by their combined span.

tIoU ≥ 0.75 as an audited target

Our audited quality target for temporal boundaries is tIoU ≥ 0.75 against reviewer reference segments, measured per batch. It's a target we report against, not a guarantee. Clips that fall short are returned, and the boundary rule that caused the disagreement is clarified in the guideline.

Reviewers also check verb–noun correctness, contact states, keypoint visibility values and object ID continuity. Errors are logged by type so trends show up early.

These rules were applied across 1M+ frames in the factory egocentric case study.

Delivery

Delivery formats

Egocentric projects usually need temporal and spatial labels in one delivery, kept in sync by shared object IDs and frame indices.

Supported formats

Ego4D-style Temporal Action JSON for actions and phases, COCO JSON for boxes, masks and hand keypoints, and CVAT XML for tracks with per-frame attributes. Custom schemas are agreed during calibration and tested against your loader before production starts.

Inspect Sample Deliverable & QA Rubric

Illustrative sample

{
  "info": "Illustrative sample — not client data",
  "video_uid": "sample_ego_001",
  "actions": [{
    "verb": "pick_up", "noun": "torque_wrench", "hand": "right",
    "pre_frame": 360, "pnr_frame": 372, "post_frame": 384,
    "start_frame": 360, "end_frame": 396
  }]
}

Stage 1 — Annotator self-check

Hand contact state set on every labelled frame
21 hand keypoints with a visibility value each
Verb–noun pairs from the closed vocabulary only
Phase frames follow the written visual cues

Stage 2 — Independent reviewer audit

Independent playback review of sampled clips
Temporal boundaries audited against tIoU ≥ 0.75 target
Object IDs checked across occlusion and re-entry
Failed batches returned for rework before delivery

Egocentric annotation technical FAQ

How is the point of no return defined for an action?

It's the first frame where the action is committed and its effect begins to show, such as a cap starting to turn or a cut starting to separate material. Each verb gets a written visual cue for it, and reviewers check that cue on playback rather than on single frames.

How do you label hand keypoints when fingers are hidden behind an object?

Each of the 21 points per hand has a visibility value. Points hidden but clearly inferable from the hand pose are placed and flagged occluded. Points that can't be inferred stay unlabelled, so the model never learns from guesses marked as visible.

What does tIoU ≥ 0.75 mean and how is it audited?

Temporal IoU is the overlap between a delivered action segment and the reviewer's reference segment, divided by their combined span. We audit sampled clips in every batch against a tIoU ≥ 0.75 target. It's a quality target we report on, not a guarantee, and batches that fall short are reworked.

How do you keep verb and noun labels consistent across annotators?

Verbs and nouns come from a closed vocabulary with written definitions and example clips. Verbs are defined by motion and effect, and nouns map to the same object classes used in the spatial labels. Additions go through a change request instead of being invented mid-batch.

How is motion blur from head movement handled?

Blurred frames get a blur attribute. Geometry follows the object's visible extent, not the smear. If an object is unreadable, annotators carry its identity from nearby sharp frames and mark the frame as low confidence rather than skipping it or guessing a new class.

Test us on your hardest first-person clips.

Start with a Free Calibration Pilot