How images become spatial answers—and what the numbers mean

Introduction

A photograph can show a chair beside a desk without telling us whether their nearest surfaces are ten centimeters or a meter apart. Answering that question requires more than recognizing two objects. The system must connect image regions to spatial evidence, put that evidence into compatible coordinates, and choose what “apart” means.

AI systems make those connections in different places. Some expose depth maps, camera poses and object points that a program can inspect and calculate with. [1, 6, 10] Others learn from geometric supervision and later answer directly from images and language. [18, 19] Across both routes, depth determines location along a viewing direction, camera alignment connects viewpoints, scale gives lengths their units, and object grounding determines which evidence a relation refers to. Systems can divide these interdependent responsibilities between explicit calculations and learned representations.

Following these connections makes a spatial answer easier to interpret. We will first see how a pixel acquires coordinates, then follow those coordinates into named objects and numerical readouts. Scale and reference frames explain what those readouts mean; geometry-supervised language models show how some of the same work can move into learned representations. The payoff is a practical distinction: a correct calculation, a plausible estimate and an independently anchored measurement can produce similar-looking answers while carrying different evidence.

A pixel becomes a point when depth selects a location

In an ideal perspective camera, an image coordinate records a ratio. A point twice as far away and twice as far to the right projects to the same pixel. OpenCV's camera model expresses this as

Here, $(X,Y,Z)$ is a point in camera coordinates; $f_x,f_y$ are focal lengths in pixels; and $(c_x,c_y)$ is the principal point. The pixel constrains a viewing ray. If optical-axis depth $Z$ is supplied, algebra reverses the projection:

This inversion assumes the relevant lens distortion has been handled and the intrinsics correspond to the image being used. Resizing or cropping an image changes its pixel-coordinate calibration. An undistorted pixel without depth remains a bearing, not a recovered three-dimensional location. [1]

The convention behind “depth” matters. With $a=(u-c_x)/f_x$ and $b=(v-c_y)/f_y$, the point is $Z(a,b,1)$, whose straight-line distance from the camera is $Z\sqrt{a^2+b^2+1}$. [1] Optical-axis depth and distance along a unit ray therefore differ away from the image center.

Stereo can constrain the missing coordinate through another observation. For ideal rectified horizontal stereo with the appropriate principal-point convention,

where $d$ is pixel disparity and $B$ is the known camera separation. [1, 2] An illustrative calculation with $f=800$ pixels, $B=0.10$ meters and $d=40$ pixels gives $Z=2$ meters. The baseline supplies the physical length unit. Finding the same target in both images is part of the operation: insufficient texture, ambiguous patterns and foreground/background matching windows can corrupt disparity. [2]

A monocular network instead learns a mapping from appearance to depth-related quantities. Depth Anything V2's October 2024 revision illustrates how supervision shapes that mapping: a teacher trained on precise synthetic labels generates pseudo-labels for real images, and students learn from those pseudo-labeled images. [3] Its basic output is affine-invariant inverse depth—relative inverse depth whose scale and offset are undetermined; metric-depth models require separate fine-tuning. [3] Feeding the basic relative output directly into the $Z$ slot above would treat it as axial depth in physical units.

The same issue appears concretely in reconstruction software. SpatialClaw's DA3 wrapper exposes an asynchronous reconstruct(video_frames, frame_indices) entry point. It reaches _reconstruct through a locked helper, runs the depth/camera model, extracts depth and intrinsics, and inverts the predicted world-to-camera matrices to obtain camera-to-world transforms. Within its per-frame loop, this exact excerpt converts the depth grid into world points: [8]

# Camera-space coordinates
x_cam = (uu - cx) * d / fx
y_cam = (vv - cy) * d / fy
z_cam = d

# Stack to (H*W, 3) and transform to world
cam_pts = np.stack([x_cam, y_cam, z_cam], axis=-1).reshape(-1, 3)  # (H*W, 3)
ones = np.ones((cam_pts.shape[0], 1), dtype=np.float64)
homo = np.hstack([cam_pts, ones])  # (H*W, 4)
world_pts = (c2w[i] @ homo.T).T[:, :3]  # (H*W, 3)
points[i] = world_pts.reshape(H, W, 3).astype(np.float32)

uu,vv identify pixels, d is this frame's depth, and c2w[i] moves camera-frame points into a shared world frame. The output retains one three-coordinate point per pixel rather than losing the image-to-point correspondence. The wrapper returns those maps together with camera-to-world poses, confidence and frame indices. It treats the nested model's depth as already in meters; that is the wrapper's input contract, not an independent physical calibration performed by this arithmetic. [8]

For an illustrative input, let an undistorted pixel be $(720,360)$, the principal point $(640,360)$, $f_x=f_y=800$, and axial depth 2 meters. The camera point is $(0.2,0,2)$ meters. With an identity camera-to-world transform, the world point stays the same. A different camera pose changes its world coordinates, not which pixel supplied it. These values are schematic inputs, not a recorded inference.

One further choice precedes even that calculation: which physical surface should the pixel represent? A glass window can show a scene behind it; a mirror can show reflected objects. DUSt3R's pointmap definition explicitly assumes a single three-dimensional hit per ray and excludes translucent surfaces. [6] A coordinate field can therefore be well formed while representing a different surface from the one the question asks about.

Viewpoints become useful together through a shared coordinate system

Two photographs put the same chair at different pixel locations. To combine their evidence, a system needs to establish how their observations refer to one scene. A camera pose supplies a rotation and translation between coordinate frames; correspondence supplies evidence that observations concern the same scene point.

COLMAP's documented route detects and matches image features, verifies matches geometrically, and incrementally reconstructs cameras and sparse points. It begins with an image pair, registers further images and triangulates new points. Recovered cameras then support denser depth/normal estimation and fusion. Its capture guidance calls for textured images, substantial overlap and viewpoints with translation, which provide useful constraints for that route. [5]

Learned reconstruction rearranges these duties. DUSt3R processes two RGB images jointly and predicts a pointmap for each, both expressed in the first input camera's frame. Neither camera intrinsics nor poses are required as inputs. For a larger image collection, however, each pair initially has its own frame and scale. Its global alignment optimizes pair-specific transforms and scales to bring repeated predictions of three-dimensional points together, weighted by confidence, rather than fitting their projections to two-dimensional image locations as ordinary bundle adjustment does. [6]

VGGT instead reasons over an image collection using alternating within-frame and global attention. It predicts cameras, depths and pointmaps, with the first camera serving as the world reference; tracking features support a query-conditioned correspondence module. A collection-wide output reduces the need to assemble separate pairwise predictions, but the different heads are still predictions. In its ETH3D reconstruction experiment, the authors obtain more accurate points by unprojecting predicted depth with predicted cameras than by using the dedicated pointmap head. [7]

That result reveals why a shared frame and geometric agreement are separate achievements. DUSt3R explicitly imposes no camera-geometry constraint on its raw network outputs, which need not fit a physically plausible projective camera. [6] VGGT reports failures with substantial non-rigid deformation. [7] A shared frame makes observations comparable; agreement with the scene still depends on the predicted geometry and its alignment.

Object names must resolve to spatial evidence before a distance can be calculated

A pointmap contains surfaces without deciding which points belong to “the chair.” An instance mask supplies that association in an image: select the pointmap entries under the mask, and the selected pixels become an object-specific point cloud. The resulting cloud contains observed or predicted visible surfaces, not automatically the complete occupied shape.

SpatialClaw's PerFrameMask.get_centroid_3d shows exactly how a cloud becomes a representative position. It resolves an absolute frame index, selects the points under an object's mask, and returns NaNs if the selection is empty. [10] For a nonempty selection, the decisive excerpt is:

# Filter by reconstruction confidence
conf = recon.points.confidence[local_r][mask_2d]
valid = conf > conf_threshold
if valid.sum() > 0:
    pts = pts[valid]
return np.median(pts, axis=0)

Despite the method name, the position is a coordinatewise median, not an arithmetic centroid or a recovered whole-object center. With the default threshold, points above confidence 0.3 are used if any pass. [10] If none pass, the function still returns a median, using all selected points. The method's string lookup also chooses the first matching label; an integer index selects a particular object. [10] Both details matter before comparing two positions.

The documented consumer takes two such positions and passes them to tools.Geometry.euclidean_distance. [11] ToolsModule installs GeometryUtils under that name, and its distance method returns the norm of the difference after checking matching shapes. [11, 12]

Consider the illustrative positions chair=(0,0,2) and desk=(0.6,0,2.8) in one frame and one length unit. Their displacement is $(0.6,0,0.8)$, and its Euclidean norm is 1. The calculation is exact for those inputs. Large nearly touching objects and tiny separated objects can have those same representative positions, so the result does not establish surface clearance.

S-Agent 2.0 makes a different part of this operation explicit: it gives later commands names to resolve. Its command-line segmentation path submits prompts and produces mask artifacts. [13, 14] At readout time, the gateway's _scene_registry reads the newest masks against the stored reconstruction, selects finite points and skips candidates with fewer than 20 points. [14] It groups matching normalized labels whose representative positions lie within a configured radius, using ground-plane separation when gravity is available. [14, 15] It then creates unique names such as chair and chair_2. These are practical association heuristics, including the possibility of grouping candidates from the same frame, rather than verified identity labels. [14]

The distinction between stored evidence and a reconstructed view matters here. scene_state.json retains reconstruction/frame references, camera poses, manually registered instances and other scene state. [15] Segmented instances are freshly derived from mask artifacts on each registry read; re-segmentation or dropping a mask can change subsequent lookups. [14] A new reconstruction initializes a new scene because its coordinates define a new address system. [14] The language agent receives names and readouts derived from this evidence, rather than every segmented object being a permanently fixed coordinate record.

Now follow an ordinary distance command. The exact CLI branch constructs this request: [13]

elif args.cmd == "distance":
    body = {"a": args.a, "b": args.b}
    if args.known is not None:
        body["known_distance"] = args.known
    body.update(request_kwargs())
    result = call("/distance", body)

call posts the payload to the gateway and returns decoded JSON. /distance builds the fresh registry, resolves the two names, subtracts their registered positions and takes the Euclidean norm. Unless a direct API request supplies another mode, it uses centroid mode: registered-position distance. With gravity, each occupied ground-footprint cell contributes one vote to a median ground position, combined with median height; without gravity, the position is a coordinatewise median. The gateway applies the stored scene scale and rounds the reported distance. [13, 14, 15]

The following is schematic I/O, not an executed command or literal complete response. Assume the registry contains the chair and desk positions above, the scale factor is one, and no gravity decomposition is available:

Command: perception.py distance --a chair --b desk
Core request: {"a": "chair", "b": "desk"}
Selected response fields:
  a: chair
  b: desk
  distance: 1.0
  note: straight-line 3D distance between the two instances' registered positions

The actual response also includes status and a units note. [14]

Positions, clouds, boxes and graph edges answer different questions

The representation determines which quantity a program can read out. Representative positions give position distance; sampled clouds give the specified cloud separation; boxes summarize estimated extents; graph edges encode inferred semantic relationships. The following operations make those distinctions concrete.

The gateway also accepts direct API modes closest and farthest, which consume reconstructed clouds rather than just positions. Its closest calculation subsamples each cloud to at most 2,500 points, computes nearest-neighbor distances in both directions, takes each side's 2.5th percentile, and returns the larger of those values. [15] The low percentile summarizes the small-distance end of each list rather than taking its minimum; the larger directional summary governs the readout. This quantity differs from the exact minimum distance between even the sampled clouds, and farther still from the minimum gap between complete physical objects. Cameras and registered spots without a cloud contribute a single point. [14]

The consumer's dispatch can change the answer's meaning. The inspected CLI has no distance --mode argument and sends none in the excerpt above. [13] In a selected S-Agent paper example, the question asks for the closest points of objects, but the displayed ordinary distance calls do not request a cloud mode. [22] At the inspected code revision, those payloads select registered-position distance. [14] This bounds the closest-surface interpretation of that displayed example; it does not establish which checkout every benchmark trajectory used or invalidate the paper's aggregate scores.

Object extents offer another useful summary. SpatialRGPT constructs axis-aligned boxes from canonicalized, denoised object points and uses boxes and point-cloud centroids to calculate training relations. [17] Its appendix explains why oriented boxes estimated from incomplete, noisy single-view clouds can also be inaccurate. [17] Changing the box type does not supply the unseen back of an object. A box is an enclosure of the available estimate, whose scene-axis dimensions can change as an object rotates.

Richer predicates require richer inference. ConceptGraphs consumes posed RGB-D, associates masked clouds using geometric and visual similarity, and fuses matched observations. [16] It then selects candidate graph edges using box overlap and a minimum spanning tree, and asks an LLM to infer likely relationships such as “on” or “in” from captions and locations. [16] The graph can store useful semantic hypotheses, but that edge-generation procedure does not measure contact or load-bearing support. A missing edge in its pruned graph is likewise not proof that a relationship is absent.

Meters need a source of scale

The examples so far become physical distances only after their length unit is justified. For ideal perspective observations, multiplying scene coordinates and camera translations by the same positive factor preserves the projections. [1] Camera calibration specifies how rays project; it does not, by itself, resolve this global scale ambiguity. VGGT makes a canonical choice by normalizing its training geometry by average point distance, then learns to predict that convention. [7] Such normalization selects a representation, not a scene-measured meter.

A known stereo baseline supplies an observed physical constraint. [2] A monocular metric-depth network supplies a learned estimate in physical units. UniDepth, for example, conditions depth on a camera representation predicted from RGB or supplied externally. [4] Its own evaluation reports cases with stronger scale-invariant geometry but incorrect scene scale on ETH3D and IBims-1, and additional scale-sensitive errors from predicted cameras on out-of-domain camera types. [4] Shape quality and meter accuracy can therefore move differently even within one model.

S-Agent exposes a third operation: accept a stated measurement as a scale anchor. If known_distance accompanies /distance, _apply_scene_calibration computes

It requires a finite positive supplied value and rejects anchor separations below 0.05 in the reconstruction's raw, pre-calibration length units. [14] It stores the factor and provenance in scene state; later readouts multiply raw quantities by that factor. A later calibration replaces the old factor rather than multiplying it again. Without calibration, its units note describes meters as the reconstruction's own uncalibrated metric scale. [14]

Illustratively, a raw anchor separation of 2 and a stated separation of 3 meters produce $s=1.5$. Another raw separation of 4 is then reported as 6 meters. The coordinates need not be rewritten: the readout applies the factor. [14] This deduction from the code assumes the anchor corresponds to the intended entities and distance definition. A global scalar can correct uniform scale error; it cannot repair a chair merged with another chair, a locally distorted cloud or an unseen surface.

This gives “metric” a useful question to answer: are the meters tied to a physical reference in this scene, statistically predicted from training experience, or inherited from an arbitrary reconstruction convention? The arithmetic can carry any of them, but they support different claims about the world.

Direction needs an observer and an orientation

Distance survives a rigid change of coordinate frame; coordinate components change. To say that the desk is “right of” the chair, a system must additionally choose whose right. A viewer's right, the chair's intrinsic facing direction and a stationary room axis need not agree. An object position alone supplies no intrinsic front.

Gravity helps establish up and down. It does not establish the chair's facing direction or geographic north. SpatialClaw's reconstruction tool tries to estimate a ground plane from a first-frame floor/ground mask and RANSAC, then rotates points and camera poses together. [9] If that route fails, it uses the first camera's down direction. [9] Its ground-height fallback uses a low percentile of visible points. [9] A coordinate system described as gravity-aligned can thus inherit an image-based estimate or a camera-orientation assumption rather than an observed floor.

The two S-Agent directional readouts make frame selection especially visible. /depth takes the signed projection of target-minus-observer onto the observer's original forward axis. For a camera, that is optical-axis depth. /side, when the observer is a camera, instead obtains gravity and constructs a ground-heading frame with this exact helper: [14]

def _ground_axes(axes: Dict[str, Any], down: np.ndarray) -> Dict[str, np.ndarray]:
    """An observer's gravity-aligned ground frame: forward projected onto the
    ground (image-up fallback for a near-vertical view), up = world up."""
    fwd_g = axes["forward_w"] - down * float(axes["forward_w"] @ down)
    if float(np.linalg.norm(fwd_g)) < 0.2:
        fwd_g = axes["up_w"] - down * float(axes["up_w"] @ down)
    fwd_g = fwd_g / np.linalg.norm(fwd_g)
    right_g = np.cross(down, fwd_g)
    return {
        "forward_w": fwd_g,
        "right_w": right_g / np.linalg.norm(right_g),
        "up_w": -down,
    }

The first subtraction removes the downward component of forward. The near-vertical branch uses projected image up because a nearly vertical forward vector provides little ground heading. The returned axes are then consumed by /side: dot products yield forward, right and up components, scene scale supplies their lengths, and thresholds of ±0.05 classify front/behind, left/right and above/below. [14] An oriented non-camera observer retains its registered axes instead of being made upright. [14]

Consequently, optical /depth and ground-heading /side can legitimately disagree for a pitched camera. As a schematic example, use right-positive X, up-positive Y and ground-forward-positive Z. Let a camera look 45 degrees downward, with forward $(0,-1/\sqrt{2},1/\sqrt{2})$, and let a target offset be $(0,-1,-0.2)$ meters. Its optical forward component is about +0.57 meters, while its ground-forward component is −0.2 meters. It is ahead along the tilted optical axis and behind in the floor-plan heading. This is an illustrative vector calculation, not a recovered scene or reported trial.

The disagreement teaches what the readouts mean. “Front” becomes computable after the system selects an observer and constructs the relevant axes. A fluent answer that leaves those choices implicit can resolve the wrong directional question even when its dot products are correct.

Geometry can teach a model without remaining an external inference input

The exposed representations above let a program inspect points and calculate quantities. Another route uses those quantities to teach the image-to-answer mapping itself.

SpatialVLM generates training questions through a pipeline involving object detection, metric-depth estimation, segmentation and descriptions. Geometric functions over object points and boxes supply answers. Its direct inference task then takes an image and a question and produces text without external tools. The geometric teacher has influenced the weights; a new question need not trigger an exposed reconstruction or scene graph. Its center-distance and gap question templates are separate, although the template appendix does not define an exact complete-surface gap operator. [18]

SpatioLM's August 2026 preprint moves supervision closer to the representation. It freezes a pretrained VLM and trains a side module whose geometry-aware features enter language-model blocks. [19] Training combines text loss, matching the side module's feature relationships to a geometric teacher's, and supervision from pseudo depth and camera rays. [19] At inference, image/video and text suffice; external depth and camera inputs are not required. [19] The paper also acknowledges inherited pseudo-label noise and a trade-off with some motion-intensive tasks. [19] Here, geometry changes the features available to the language model instead of supplying a numerical tool to an otherwise unchanged agent.

Geometric supervision can thus remain useful after its external inputs disappear from the inference interface. Learned spatial expectations and inspectable calculations offer complementary ways to connect images to an answer.

A useful spatial answer needs evidence matched to its quantity

Evaluation can now ask a more precise question than whether a model “understands space.” Ordering two objects by depth, estimating their representative-position distance, finding surface clearance and maintaining a map as the observer moves test different operations.

SpatialVLM's quantitative evaluation, for example, counts estimates between half and twice a human reference as in range. [18] Its authors connect its useful distance range to the depth teacher's biases. [18] That experiment supports a capacity for approximate spatial estimation under its protocol, not precise clearance measurement.

Even a correct ordering can come from a visual shortcut. Why Far Looks Up separates cases where image height agrees with depth from counterexamples and constructs a synthetic benchmark that manipulates height independently. [20] Its evaluated Molmo, NVILA and Qwen2.5 families can improve average performance while retaining shortcut dependence. [20] The same study reports much smaller consistent/counterexample gaps for reasoning-enabled GPT-5.2 and Gemini-2.5-Pro under a final-output exact-match protocol. [20] Breaking the usual link between image height and depth reveals how much a tested model relies on that cue.

Confidence also has an operational meaning. UniDepthV2's December 2025 revision evaluates whether predicted uncertainty puts more erroneous pixels ahead of accurate ones when ranking pixels for removal. [21] For its Large model, normalized sparsification error rises from 0.199 in-domain to 0.645 on unseen datasets, while Spearman rank correlation falls from 0.744 to 0.299. [21] In this metric, zero normalized sparsification error is oracle ranking and one is random ranking. [21] The deterioration under domain shift concerns pixel-error ranking, not the probability that an object's clearance exceeds a required margin.

Producing such an object-level interval would require accounting for more than pixel-depth error: mask identity, missing extent, scale and frame estimates all enter the final relation. A practical spatial answer should make its quantity and evidence legible—for example, “registered-position distance on the reconstruction's uncalibrated scale,” or “right in camera 3's gravity-aligned heading.” These descriptions tell a later consumer which calculation was supported and what additional observation might change it.

Conclusion

AI turns images into spatial answers by connecting visible evidence and learned expectations to quantities with defined meanings. Depth or point prediction locates image evidence; camera alignment makes views comparable; segmentation and association connect points to objects. Scale determines the physical meaning of length, while an observer and orientation determine direction. A representative position, visible cloud, bounding box and semantic graph edge answer different questions about the same object.

Those responsibilities can be exposed as inspectable geometry and code, learned into a model's representations, or divided between the two. The important consequence is that numerical exactness begins only after the inputs and definitions have been chosen. A norm can be computed perfectly from the wrong object, and a coherent shape can carry the wrong meter scale. The next useful question is therefore specific: which part of this answer came from observations of this scene, which part was supplied or learned, and what missing evidence could change the relation?

Limitations

This account explains selected primary methods and static implementations available through November 2, 2026. External project code, inference and benchmarks were not executed. The illustrative coordinates, requests and vector calculations are teaching examples, not measured outputs. Source inspection establishes the computations and conditional branches at the pinned revisions, not their deployment frequency or new-scene accuracy.

The implementation examples are not a single interchangeable stack: standalone SpatialClaw and S-Agent have distinct revisions and interfaces. S-Agent's inherited perception dependency is not independent validation of SpatialClaw's geometry. The selected paper trajectory cannot establish the exact source checkout used in every experiment. No general model ranking or independently reproduced performance claim follows from this comparison.

The evidence does not establish complete hidden-object geometry, reliable identity association in arbitrary scenes, or calibrated uncertainty for final object relations and actions. The cited studies use different input regimes, supervision, alignment conventions and evaluation tolerances. Their results support the local distinctions explained here, not a safety guarantee for navigation or manipulation.