Dataset Construction Pipeline

Stage 1

Scene Point-Cloud Collection and Semantic Fusion

Build a scene-level geometric backbone aligned with RGB, segmentation, LiDAR, and pose data.

Stage 1 starts from the simulator scene itself together with the runtime configuration that defines sensor settings, sampling bounds, altitude, and pose coverage. The goal is not to generate benchmark questions directly, but to reconstruct a stable scene-level geometric backbone that later stages can mine repeatedly without returning to image space.

In the current implementation, this stage follows the same logic described in the paper: multi-pose sampling, raw chunk capture, and global semantic fusion. RGB, segmentation, LiDAR, and pose metadata are captured at every sampled pose, then transformed into a unified scene coordinate system so that semantic and instance identities remain attached to the fused point cloud. The result is a scene-scale representation that Stage 2 can split directly into landmark candidates rather than re-detecting objects from scratch.

The key output of Stage 1 is therefore not a benchmark row but a geometric substrate. By the time this stage finishes, every released scene already has a stable map boundary, fused point cloud, global coordinate system, and scan preview record. That shared spatial backbone is what lets later stages reason about landmark visibility, relative position, orbit feasibility, and scene coverage without repeatedly solving low-level mapping problems.

The scene scanning library below exposes benchmark-facing panoramas for the released test scenes while keeping Stage 1 metadata visible. Each entry now combines two released RGB scene snapshots with two segmented point-cloud panoramas rendered in top-down and oblique views, together with the configured scene boundary, mapped area, and the complete unscreened landmark-candidate count that Stage 2 begins from.

ENV 7 · Waterfront City

Two RGB scene snapshots + two segmented point-cloud panoramas + Stage 2 raw candidate pool

Scene Boundary x[-219, 191] · y[-270, 268] · z[-50, 52]

Mapped Area 220580 m²

Raw Landmark Candidates 167

Reviewed Valid Landmarks 86

Previous 1 / 12 Auto-play: On Next

Scene Scanning Library. Two benchmark-facing RGB scene snapshots and two segmented point-cloud panoramas are shown for each released test environment together with Stage 1 boundary metadata and the unscreened Stage 2 candidate pool.

Stage 2

Landmark Review and Semantic Annotation

Turn the fused scene cloud into a reviewed, semantically named landmark pool.

Stage 2 consumes the fused semantic and instance-aware cloud from Stage 1 and converts it into reviewed landmark assets. The pipeline first groups points by semantic class and instance identity, then renders multiview RGB and point-cloud evidence around each candidate so that geometric structure, appearance, and view direction are all available before any benchmark task is built.

For each landmark candidate, the pipeline computes eight orbit-view capture poses from the 3D bounding box and instance center, covering the canonical directions front, front-right, right, back-right, back, back-left, left, and front-left. The associated segmented point-cloud views are then used to perform occlusion-aware validity checks, so heavily blocked or non-informative directions can be excluded before benchmark-facing landmark assets are finalized.

This is the stage where scene geometry becomes benchmark object assets. Review removes unstable or uninformative candidates, confirms the main view, and finalizes the reviewed landmark manifest used by both Stage 3 and Stage 4. After review, the pipeline runs a constrained auto-label prompt to add coarse category, fine-grained subcategory, and a short discriminative description that later prompts can reuse.

The important shift is that Stage 2 turns a fused but still machine-oriented scene map into benchmark-facing semantic objects. Candidate aggregation, multiview review, and prompt-constrained annotation are linked in one chain: only after geometry, visibility, and main-view consistency are confirmed does the pipeline allow a landmark to receive the semantic fields that later prompt templates depend on.

Step 1 · Candidate Aggregation. The pipeline first partitions the fused semantic cloud by semantic class and instance identity. Each resulting candidate is then packaged with multiview RGB evidence, point-cloud views, 3D center metadata, and 3D box support so that later review decisions are made against geometry-aware evidence rather than isolated screenshots. At this stage the pool is intentionally permissive: the goal is to surface all plausible landmark candidates before benchmark-facing filtering begins.

Step 2 · Review and Main-View Confirmation. Human review then removes unstable or uninformative instances, confirms the most representative main view, verifies one landmark-centric direction anchor, and checks bbox validity. The main view is not required to be the front image; it is simply the clearest and most benchmark-facing reference view. Once that anchor direction is confirmed, the remaining seven directions are derived automatically in the fixed orbit order rather than being edited independently. At this step, reviewers also manually remove unusable view images that remain visually ambiguous, severely occluded, or otherwise unsuitable even after the earlier geometric filtering. Candidates can be dropped immediately, but a landmark is only kept after the main-view and direction checks are complete. The accepted pool is frozen in the reviewed landmark manifest (valid_instances.json), which becomes the only stable Stage 2 handoff consumed by both later stages. In practice this step turns a broad candidate inventory into a controlled landmark repository whose direction labels, representative views, and spatial references can be trusted downstream.

Step 3 · Prompt-Constrained Semantic Annotation. After geometric review, the pipeline runs a constrained auto-label prompt to enrich each landmark with benchmark- facing semantics. This is not free-form captioning: the prompt explicitly constrains category choices, subtype naming, and JSON formatting so that landmark semantics remain short, discriminative, and machine-parseable. The result is a semantic layer that later prompt templates can reuse directly without re-describing each landmark from scratch.

In practice, Stage 2 Step 2-4 are completed in the internal review web. That interface is where reviewers perform candidate screening, main-view confirmation, single-direction anchoring, invalid-view cleanup, auto-label launch, and final semantic audit. The command line remains useful for collection and standalone auto-label reruns, but the web is the recommended working surface whenever view quality and semantic correctness need to be judged together from visual evidence.

The reviewed landmark scanning library below makes that Stage 2 handoff concrete: it shows each retained landmark view together with the paired segmented point-cloud rendering captured from the same scanning pose. This is the evidence basis from which later prompt templates, mission planners, and image-task constructors all reuse Stage 2 landmark assets after review and semantic labeling are finalized.

Landmark Mid Rise Building · ENV 7High Rise Building · ENV 16Prefabricated Container Building · ENV 9High Rise Building · ENV 11Wooden Scaffolding Structure · ENV 20Mid Rise Building · ENV 10Mid Rise Residential Building · ENV 7Mid Rise Building · ENV 7Small Shed · ENV 9Sedan Car · ENV 23

Previous Auto-play: On Next

Mid Rise Building

Building · Mid Rise Building · ENV 7

light gray mid-rise building with red tiled roof and dormer windows

RGB

Point Cloud

RGB

Point Cloud

RGB

Point Cloud

Occluded / Invalid

Occluded / Invalid

RGB

Point Cloud

Occluded / Invalid

Occluded / Invalid

RGB

Point Cloud

RGB

Point Cloud

Landmark Scanning Library. Reviewed landmark-centric RGB side views are shown together with their corresponding segmented point-cloud views, preserving the scan-level evidence that underlies later image and video task construction.

Stage 3

Behavior-Driven Video Task Generation

Compose missions, repair trajectories, render benchmark videos, and export video manifests and metrics.

Stage 3 begins from reviewed Stage 2 landmarks and an explicit two-layer behavior system. In the current single-landmark release, each valid landmark is bound to one atomic mission and one composite mission. Those missions are not just labels: they expand into executable trajectories, camera-control programs, temporal supervision tracks, and final benchmark manifest rows.

The flight-mode library is designed as a benchmark primitive rather than a loose collection of cinematic motions. The intent is to keep released trajectories close to common UAV operating scenarios, so that the resulting video tasks reflect recognizable drone behavior patterns instead of synthetic camera sweeps with little operational meaning. At the low level, the atomic motions borrow from the design logic behind DJI's automated “MasterShots”-style capture patterns, where approach, orbit, rise, and mapping maneuvers are defined as reusable camera-flight units. At the high level, the hierarchical flight-mode design organizes composite classes around common inspection and mapping needs, so that the benchmark's temporal reasoning tasks remain tied to realistic patrol, inspection, and surveying workflows.

Behavior Hierarchy. Composite inspection classes are instantiated by chaining atomic maneuvers, and the highlighted links show which primitives each composite template reuses in the released Stage 3 library.

The implemented chain is mission generation, trajectory search, trajectory repair, video recording, and temporal task organization. Collision checks and safety buffers can trigger radius, height, or scan-width adjustments before a trajectory is accepted, so the final videos are filtered task assets rather than naïve recordings of an idealized path. Parallel rendering is also scheduled to keep missions spatially separated and reduce multi-UAV contamination in the frame.

Once rendering succeeds, Stage 3 exports both mission-level supervision bundles and the benchmark-facing datasets used for Flight Behavior Recognition and Temporal Localization and Landmark Visibility Counting and Interval Reasoning. Archived 4K frame captures and released 1080P H.264 MP4s are separated on purpose, which lets the benchmark keep high-resolution supervision upstream while distributing efficient evaluation videos downstream.

In other words, Stage 3 is where motion programs become auditable temporal tasks. The stage has to satisfy two goals at once: trajectories must remain executable under collision and safety constraints, and the final media must preserve enough temporal structure that interval localization remains meaningful for large multimodal models. That is why mission repair, behavior templating, and supervision export are treated as one connected pipeline rather than three independent utilities.

The media contract used at this stage is therefore worth exposing directly. Internally, missions still retain higher-resolution archived frame captures for supervision and refresh, while the released benchmark distributes a lighter 1080P H.264 video stream at a fixed frame rate. This separation keeps temporal evaluation efficient for public use while preserving enough upstream visual detail to support rerendering, inspection, and higher-resolution downstream analysis when needed.

The Stage 3 internal web is used here as an interactive mission and task workbench. It exposes pages for the behavior library, mission generation, candidate review, manifest browsing, experiments, result inspection, and metrics. This makes it well suited for qualitative verification and targeted reruns, whereas task_pipeline.py remains the recommended interface for batch selection, data export, render, and released-scale experiment phases.

Stage 3 Media Specs. Mission rendering retains 4K-grade archived frame captures upstream, while the released benchmark exposes 1080P H.264 videos at a fixed 10 FPS evaluation contract.

Behavior Library. The released Stage 3 defaults expose the composite set templates, atomic building blocks, and parameter defaults that drive mission generation.

Stage 4

Image QA Generation, Render-Only Asset Refresh, and Evaluation

Sample structured image QA rows and keep assets refreshable without changing task semantics.

Stage 4 reuses the reviewed Stage 2 landmark pool, but its target product is structured image QA rather than temporal supervision. In the current single-landmark core release, every valid landmark contributes four task families under two difficulty settings, which yields a fixed 512 × 2 × 4 = 4096-sample lattice before experiments begin.

The image-task library is designed to make dual cognition visible at the level of question formulation rather than only at the level of evaluation metrics. The self-aware branch asks where the UAV is and what it will see after a described motion, while the environment-aware branch asks where the target lies and what action is appropriate under the current landmark-relative situation. This split ensures that image tasks are not just generic multiple-choice perception items: they are built to separate self-state reasoning from target-oriented situational reasoning under the same landmark-centered world model.

The image branch is not assembled by randomly pairing screenshots. Landmark-Relative Position Reasoning and Future Observation Prediction share landmark-centric reference families but ask different questions about current position and motion-induced future observation. Self-Relative Position Reasoning and Landmark-Driven Action Decision share the same egocentric observation so that target direction reasoning and action decision are grounded in identical evidence. For the environment-aware branch, the pipeline repeatedly resamples observation poses and camera yaw until the landmark stays fully visible and its final bbox is valid.

Stage 4 therefore exports both semantic QA rows and rendering sidecars. The manifests keep the benchmark interface stable, while rerender request files preserve the capture recipe that allows assets to be refreshed later without changing sample ids, options, or answers.

This stage is also where the image benchmark becomes deliberately paired and evidence aware. Reference images, query observations, answer options, and normalized bbox targets are emitted together so that semantic selection and spatial grounding can be evaluated on the same sample instead of in disconnected protocols.

The release-side media specification stays explicit here as well. Stage 4 keeps image assets at the same DCI 4K capture standard used upstream, preserves a uniform 4:3 aspect ratio across the branch, and applies a bounded JPEG compression policy so that public QA assets remain compact without discarding the visual detail needed for fine landmark grounding.

The Stage 4 internal web plays a similar role on the image side: it is the place to inspect generated manifests, preview task rows, launch comparison runs, browse per-sample outputs, and read metric summaries. For small interactive checks, this web surface is the fastest way to verify that reference images, query observations, option layouts, and bbox targets are aligned; for released-scale generation and experiment batches, the recommended path is still task_pipeline.py under Stage 4 selection/data/render/experiment phases.

Stage 4 Media Specs. Released image tasks preserve DCI 4K landmark renders under a uniform 4:3 aspect ratio and a JPEG 80 export policy so that semantic QA and spatial grounding share the same high-resolution evidence contract.