Bridging Language and 3D World Generation through Spatial-Graph Reasoning and Physics Constraints

Bridging Language and 3D World Generation through Spatial-Graph Reasoning and Physics Constraints

Baibhab Das

School of Computer Science and Engineering, VIT-AP University, Andhra Pradesh, India.

Abstract – Teaching machines to reason about three-dimensional environments using natural language alone is genuinely hard. Existing text- to-3D methods routinely produce scenes that look physically absurdobjects oat in mid-air, meshes intersect in impossible ways, and heavy objects rest atop imsy supports. We tackle this head-on with a unied framework that weaves together natural language understanding, spatial reasoning, and physical plausibility. At its core sits a Spatially Grounded Transformer working alongside graph neural networks. Three components do the heavy lifting: a Graph Neural Scene Reasoner (GNSR) that explicitly models object relationships, a Latent Alignment Module (LAM) that bridges reasoning and generation, and a Physics-Aware Scene Consistency Module (PSCM) that catches physical nonsense. We push free-form text through an NLP pipeline that handles synonym replacement and dependency parsing, then feed the results into these modules. The system achieves 63.0% visual question answering accuracy across CLEVR, Objaverse, ShapeNet, and Text2Shape benchmarks. More importantly, it slashes the Physics Violation Score by 87.8%, dropping it from 41% to just 5%.

Keywords: Text-to-3D, Graph Neural Networks, Latent Alignment, Spatial Reasoning, Scene Understanding, Visual Question Answering, Physics-Aware Generation, Attention Mechanisms, Diffusion Models.

1. INTRODUCTION

   Natural language processing and 3D scene understanding have each advanced considerably, but bringing them together remains stubbornly difcult. People describe visual scenes effortlesslythe red cube sits to the left of the blue sphereyet building systems that can go the other direction, synthesizing or comprehending 3D scenes from such descriptions, opens exciting possibilities in robotics, augmented reality, and autonomous systems.

   Heres the frustration: state-of-the-art text-to-image models like DALL-E and Stable Diffusion produce stunning 2D results, and single-object 3D generators like Shap-E and Point-E keep getting better. But throw multiple interacting objects at them, and the wheels come off. These systems dont reason about relationships between objects. They dont check whether a generated scene makes physical sense. The typical pipeline generates objects independently, then assembles them as an afterthought. The outcome? Floating objects, intersecting meshes, and scenes that violate basic physics.

   What we really need is a system that understands object properties and physical constraints from the ground up. Not something that generates rst and checks later, but something that builds physical plausibility into its core reasoning. Thats exactly what were proposing here.

   Heres what we bring to the table:

   1. A multi-stage NLP parser that extracts structured information from messy free-form text using regex, spaCy dependency parsing, synonym expansion, and sensible defaults when things get ambiguous.

   2. Graph Neural Scene Reasoner (GNSR): A hybrid architecture where objects become nodes in a graph and spatial rela- tionships become edgestreating scene understanding as explicit relational reasoning rather than a black-box operation.

   3. Latent Alignment Module (LAM): A differentiable bridge that aligns our scene representations with Shap-Es latent space, making the whole pipeline end-to-end trainable.

   4. Physics-Aware Scene Consistency Module (PSCM): Three carefully designed loss functions that penalize collisions, unsupported objects, and gravitational impossibilities.

   5. Real-world validation: Testing on Objaverse, ShapeNet, and Text2Shape alongside synthetic CLEVR data.

   6. Physics Violation Score (PVS): A new metric that quanties just how physically plausible a generated scene really is.

   7. Thorough evaluation: Comprehensive experiments and ablation studies that reveal what each component actually contributes.

   Underneath all this lies what we call Spatially-Grounded Relational Learning (SGRL). Three principles guide it: (1) build explicit relational graphs that capture object interactions, (2) let geometry inform attention mechanisms, and (3) keep reasoning and generation tightly coupled through differentiable alignment.

   Weve organized the paper like this. Section II surveys related work. Section III lays out the formal problem and mathematical machinery. Section IV walks through the architecture. Section V covers experiments and results.

2. RELATED WORK

   1. Text-to-Image and Text-to-3D Generation

      Diffusion models have completely changed whats possible in text-to-image generation. Rombach et al. made a clever observationrunning diffusion in compressed latent space rather than pixel space makes high-resolution synthesis tractable:

      2

      LLDM = EzE(x),EN (0,1),t [lE E (zt, t, c)l2] (1)

      E

      Here is the encoder, c is the text conditioning, and E is the denoising U-Net. Classier-free guidance, introduced by Ho and Salimans, cranks up the inuence of conditioning:

      E(zt, t, c) = E(zt, t, )+ s · (E(zt, t, c) E(zt, t, )) (2)

      with guidance scale s> 1.

      On the 3D front, early voxel-based approaches hit a scaling walltheir O(N 3) complexity made high-resolution outputs impractical. Neural radiance elds offered a more elegant path, representing scenes as continuous functions:

      F : (x, d) (c, ) (3)

      where x is a 3D coordinate, d the viewing direction, c the predicted color, and the volume density. OpenAIs Shap-E went further, generating signed distances and colors conditioned on a latent code:

      f : (x, z) (s, c) (4)

      Heres the catch, though. All these systems are fundamentally single-object generators. They dont know how to arrange multiple objects in a scene. Thats a different problem entirely, and its the one were tackling.

   2. Scene Understanding and Reasoning

      The CLEVR benchmark pushed VQA research forward by providing a controlled testbed for compositional reasoning. A CLEVR scene renders as:

      Scene = Render(O, R, L) (5)

      O R L

      where denotes objects, spatial relations, and lighting conditions. Figure 1 shows the exploratory analysis we ran on CLEVRdistributions of objects per scene, colors, shapes, materials, sizes, and question types.

      Fig. 1: Exploratory Data Analysis of the CLEVR dataset: (a) distribution of objects per scene, (b) color distribution, (c) shape distribution, (d) material distribution, (e) size distribution, and (f) question type distribution.

      The NS-VQA system combined neural perception with symbolic reasoning:

      a = SymbolicReasoner(ObjectDetector(I), ProgramGenerator(Q)) (6)

      Transformers have also made their way into scene understanding. Vision Transformers treat images as sequences of patch embeddings:

      xp = [xclass; x1; x2; … ; xN ]+ Epos (7)

      p p p

      DETR reframed detection as set prediction with bipartite matching:

      L

      N

      LHungarian(y, y) =

      i=1

      log p(i)(ci)+ Wci =Lbox(bi, b

      (i))L

      (8)

      These works convinced us that slot attention was the right tool for pulling etity representations out of text.

   3. Spatial Reasoning in NLP

      Spatial language has drawn steady attention from the NLP community. Early work used rule-based approaches; dependency parsing and automatic relation extraction gradually took over. But spatial language remains trickywords like near are famously context-dependent, and pinning down their precise meaning is harder than it looks.

      Recent work folds spatial information directly into transformer architectures, encoding pairwise spatial relationships as

      attention biases:

      aij =

      (xiWQ)(xjWK)T + (xiWQ)(rijWR)T

      d (9)

      where rij captures the relative spatial position between tokens i and j.

   4. Physical Reasoning in AI

   Theres been real progress on equipping neural models with physical intuition. Interaction networks learned how objects inuence each other through explicit pairwise interactions. Graph networks later scaled this up to simulate physical dynamics at much greater complexity. Physics-informed neural networks have also emerged as a promising direction.

   To the best of our knowledge, no prior work has combined text-to-3D generation with explicit physical consistency losses. Thats the gap were lling.

3. Problem Formulation and Mathematical Framework

   1. Preliminaries and Notation

      Lets set up the notation carefully. We have a text description T = {t1, t2,…, tL} with L tokens. The goal is a physically

      i=1

      consistent 3D scene S = {(oi, pi, R)}N , where:

      • N is the number of objects

      • oi = (ci, si, mi, zi) encodes color ci C, shape si Sh, material mi M, and size zi Z

      • pi = (xi, yi, zi) R3 gives the 3D position of object i

        i,j=1

      • R = {rij}N is the set of pairwise spatial relations drawn from Rel = {left, right, front, behind, above, below, near, far}

        A

        Given a question q, the system also needs to produce an answer a .

        S|T S|T

        M T

        Physically Consistent Neural Scene Generation (PCNSG): We dene a model that maps text to a scene distribution P( ). Standard approaches maximize max ESP [log P ( )], completely ignoring physical plausibility. PCNSG xes this with an explicit physical penalty:

        .. ..,. .,

        LPCNSG = Lrecon(S, S)

        Scene Reconstruction

        +phys Lphysics(S)

        .. ..,. .,

        Physical Consistency

        +align Lalign(zpred, ztrue)

        .. ..,. .,

        Latent Alignment

        (10)

        The physics term Lphysics has three components: collision avoidance (Lcoll), gravitational stability (Lgrav), and support propagation (Lsupp).

   2. Text Encoding with DistilBERT

      We use DistilBERT for text encodingits smaller, faster, and still gets the job done. The initial representation combines token, positional, and segment embeddings:

      H(0) = Etoken + Epos + Eseg (11)

      Each transformer layer applies multi-head self-attention followed by a feed-forward block:

      H(l) = LayerNorm(H(l1) + MultiHead(H(l1))) (12)

      H(l) = LayerNorm(H(l) + FFN(H(l))) (13)

      The encoder outputs H = H(L) RL×dt with dt = 768 over L = 6 layers. Multi-head attention works as follows:

      MultiHead(Q, K, V) = Concat(head1,…, headH )WO (14)

      headh = Attention(QWQ, KWK, VWV ) (15)

      )

      h h h

      Attention(Q, K, V) = softmax

   3. Spatial Encoding with Fourier Features

      QKT

      dk

      V (16)

      Neural networks have a well-known blind spot for high-frequency spatial signalsspectral bias, they call it. We get around this using Fourier positional encodings:

      (p) = [sin(2Bp), cos(2Bp), p] (17)

      N

      where B RF ×3 is drawn from (0, 2). Frequency bands are spaced exponentially as fk = 2k1 for k = 1, …,F . A small MLP projects the encoding to 256 dimensions:

      espatial = MLP((p)) R256 (18)

      Two linear layers with GELU activations and LayerNorm between them do the trick.

   4. Slot Attention for Entity Extraction

      )

      Slot attention extracts up to K = 8 object-level representations from the projected text embeddings X RL×d. Learnable initial slot vectors Q(0) RK×d get rened iteratively. At each step t = 1, …,T :

      A(t)

      = softmax

      Q(t1)XT

      d

      K×L

      (19)

      R

      U(t) = A(t)X RK×d (20)

      Q(t) = GRU(Q(t1), U(t)) (21)

      After T = 3 iterations, the nal slots S = Q(T ) RK×d carry object-centric representations.

   5. Graph Neural Scene Reasoner (GNSR)

      This is where things get interesting. The GNSR builds a scene graph G = (V, E ) where nodes are objects and directed edges

      capture spatial relationships (Figure 2). Node features come from slot representations: h(0) = s . Edge features e combine

      spatial and relation-type embeddings.

      v v uv

      Fig. 2: GNN reasoning through scene graph visualization. Nodes depict objects (red cube, blue sphere, green cylinder), while colored directed edges indicate spatial relationships: left-right (red), up-down (green), near-far (purple), front-back (blue).

      m = L

      Node representations update via message passing across LGNN = 3 layers:

      (l)

      v

      uN (v)

      MLPmsg(h(l1), euv) (22)

      u

      h(l) = GRU(h(l1), m(l)) (23)

      v

      v=1

      v v v

      GNN

      The nal node representations H

      = {h(3)}K

      RK×d encode both per-object attributes and inter-object structure.

   6. Spatially Grounded Self-Attention

      Heres our core innovation: augmenting self-attention with spatial bias. Starting from entity slots S and spatial embeddings

      Espatial, we compute queries, keys, and values:

      Q = SWQ, K = SWK, V = SWV (24)

      A learned spatial bias comes from the spatial embeddings:

      Bspatial = EspatialWB RK×H (25)

      This bias goes directly into the attention scores:

      Ahij =

      (Q ) (K )T

      hi: · hj:

      d/H + (Bspatial)ih (26)

      Why does this matter? Spatially close objects attend more strongly to each other, letting geometry guide relational reasoning.

   7. Latent Alignment Module (LAM)

      LAM creates a differentiable bridge between scene reasoning and Shap-Es generative latent space. An MLP maps GNN output to a predicted latent:

      zpred = MLPalign(HGNN) R512 (27)

      The alignment loss penalizes distance to the ground-truth latent:

      2

      Lalign = lzpred ztruel2 (28)

      Because gradients ow back through the reasoning modules, the entire pipelinefrom text parsing to 3D generationtrains end-to-end.

   8. Physics-Aware Scene Consistency Module (PSCM)

   PSCM enforces physical plausibility through three carefully designed losses (Figure 3).

   Fig. 3: Comparison between scenes without and with PSCM. Left: scenes that defy physicsoating objects, intersecting meshes, gravity violations. Right: physically plausible arrangements.

   Collision Loss: Objects shouldnt overlap. Using bounding sphere radii ri, we penalize interpenetration:

   N

   N

   2

   coll

   N (N 1)

   i

   j

   i

   j

   2

   L = 2 L L max(0,r + r lp p l ) (29)

   i=1 j=i+1

   Support Loss: Objects that need support shouldnt oat:

   L

   N

   Lsupp

   = 1 W

   N

   i=1

   needssupport(i)

   · max(0, zi z

   ground

   (i))2 · W

   nosupportbelow(i)

   (30)

   Gravity Loss: Heavy objects shouldnt rest on lighter ones:

   grav

   N

   N

   i

   j

   zi>zj

   L = 1 L L

   max(0,w w ) · W

   (31)

   i=1 j:j supports i

   I. Bidirectional Cross-Attention for Text-3D Fusion

   We fuse language and scene embeddings with cross-attention in both directions. Language informs the scene:

   L S : Slang = CrossAttn(S, L, L) (32)

   And the scene informs language:

   S L : Lscene = CrossAttn(L, S, S) (33)

   Residual connections and LayerNorm follow each direction:

   S1 = LayerNorm(S + Slang) (34)

   L1 = LayerNorm(L + Lscene) (35)

   J. Pairwise Relation Prediction and Coordinate Prediction

   Given fused representations F RK×d, pairwise features concatenate object representations:

   fij = ReLU(W1[fi, fj]+ b1) Rd/2 (36) Relation logits: rij = W2fij + b2 R|Rel|+1.

   Object coordinates come from a two-layer MLP with tanh output scaling:

   pi = tanh(W3 · ReLU(W4fi + b4)+ b3) · 3.0 R3 (37)

   K. Loss Functions

   Training uses a multi-task loss:

   Ltotal = Lans + 1Lcolor + 2Lshape + 3Lmaterial + 4Lsize + 5Lcoord + 6Lrel + alignLalign + physicsLphysics (38) We use cross-entropy for discrete attributes and MSE for coordinates. Weights: 1 = 2 = 3 = 4 = 0.3, 5 = 0.01,

   6 = 0.1, align = 0.1, physics = 0.5.

4. ARCHITECTURE AND WORKFLOW

   1. System Overview

      Eight tightly integrated modules form the full system (Figure 4): (1) Scene Entity Extractor using slot attention; (2) 3D Spatial Embedding Encoder with Fourier features; (3) Graph Neural Scene Reasoner; (4) Spatially Grounded Transformer; (5) Latent Alignment Module; (6) Bidirectional Text-to-3D Fusion; (7) Physics-Aware Scene Consistency Module; and (8) Scene Reasoning Module for question answering.

      Fig. 4: Full architecture of the Physically Consistent Spatially Grounded Transformer for text-to-3D scene understanding, showing all eight modules.

   2. NLP Parsing Pipeline

      The parser uses multiple passesregex rst, then spaCy dependency parsing, then a sliding window scanwith synonym expansion from WordNet. This layered strategy helps the parser recover gracefully when any single pass fails.

   3. 3D Mesh Generation Pipeline

   LJ

   Shap-E conditions on LAM-predicted latents rather than text prompts directly. For multi-object scenes, individual meshes merge:

   1. Datasets

      N

      Mscene = Mi (39)

      i=1

5. Experimental Setup

   We evaluate on CLEVR (100,000 scenes with 310 objects, 8 colors, 3 shapes, 2 materials, 2 sizes), Objaverse (50,000 annotated objects), ShapeNet (10,000 CAD models), Text2Shape (5,000 text-shape pairs), a synthetic dataset of 1,000 scenes, and a held-out test suite of 29 manually written inputs.

   We also introduce two evaluation tools: Physics Violation Score (PVS) measures the fraction of objects violating at least one physical constraint. The Text-to-Physical Scene (T2PS) Task combines attribute accuracy, relation F1, VQA accuracy, and PVS into a single benchmark.

   1. Implementation Details

      × ×

      PyTorch 2.0 implementation. Key hyperparameters: batch size 16, learning rate 2 104, weight decay 1 104, 10 epochs, dt = 768, d = 256, K = 8 slots, H = 8 heads, 4 transformer layers, 3 GNN layers. Shap-E uses guidance scale s = 15.0 and 64 diffusion steps.

   2. Training Dynamics

      Figure 5 shows training and validation loss curves alongside validation accuracy over 10 epochs. Training loss drops from

      3.59 to 2.18; validation loss from 3.11 to 2.47. Validation accuracy climbs from 39.5% to 63.0%, suggesting effective learning without overtting.

      Fig. 5: Training and validation loss curves (left) and validation accuracy (right) over 10 epochs.

   3. Evaluation Metrics

   We track VQA Accuracy, Attribute Accuracy (per attribute type), Relation F1 Score, Scene Graph Accuracy, Parsing Success Rate, average Spatial Relation Count, Physics Violation Score, and CLIP-Similarity between generated and described scenes.

6. Results and Analysis

   1. Quantitative Results

      The full model achieves 63.0% VQA accuracy, 12.4% color accuracy, 43.0% shape accuracy, 73.4% relation F1, 68.2% scene graph accuracy, 100% parsing success rate, 2.2 objects per scene on average, 3.3 relations per scene, 5% PVS, and 0.84 CLIP-similarity. Table I summarizes everything.

      Metric Performance

      VQA Accuracy 63.0%

      Color Attribute Accuracy 12.4%

      Shape Attribute Accuracy 43.0%

      Relation F1 Score 73.4%

      Scene Graph Accuracy 68.2%

      Parsing Success Rate 100%

      Avg Objects per Scene 2.2

      Avg Relations per Scene 3.3

      Physics Violation Score (PVS) 5%

      CLIP-Similarity 0.84

      TABLE I: Model Performance on CLEVR Validation Set and Real-World Datasets

   2. Physics Violation Score Analysis

      Figure 6 tracks PVS as we add modules one by one. Baseline with no spatial or physical awareness scores 41%. Adding spatial bias drops it to 32%. Adding GNN brings it to 21%. LAM reduces it to 18%. The full model with all three physical losses achieves 5%an 87.8% overall reduction.

      Fig. 6: Physics Violation Score (PVS) across model variants. Baseline achieves 41% PVS; adding spatial bias reduces to 32%;

      +GNN to 21%; +LAM to 18%; full model with physics losses achieves 5%.

   3. Confusion Matrix Analysis

      The confusion matrix in Figure 7 reveals where predictions go wrong. Color-wise, blue gets mislabeled as cyan (32% of errors) and red as orange (28%). Shape confusion clusters around cylinders and cubes (25%). Theres also a systematic under-counting bias.

      Fig. 7: Confusion matrix for VQA answer predictions. Diagonal elements are correct predictions. Errors occur between semantically similar classes.

   4. Ablation Studies

      Table II breaks down each components contribution. Removing GNN drops VQA accuracy from 63.0% to 54.1% and raises PVS to 12%. Running LAM in pipeline mode (disconnecting end-to-end gradients) costs 1.2 VQA points but hurts PVS

      more noticeably (15%). The spatial embedding encoder and spatially grounded transformer are the most criticalremoving either crashes VQA to around 41-44%. Removing Fourier features increases coordinate MSE by 23%, matching spectral bias predictions.

      TABLE II: Ablation Study Results

      Conguration

      VQA Acc

      Coord MSE

      PVS

      Full Model	63.0%	0.098	5%
      – GNN Reasoning	54.1%	0.124	12%
      – LAM (Pipeline mode)	61.8%	0.112	15%
      – Spatial Embedding Encoder	41.3%	0.187	18%
      – Spatially Grounded Transformer	44.1%	0.156	22%
      – Text-to-3D Fusion	44.1%	0.131	14%
      – Fourier Features	46.5%	0.152	8%
      – Slot Attention	42.8%	0.168	16%
      – Collision Loss	62.1%	0.102	18%
      – Support Loss	62.5%	0.105	22%
      – Gravity Loss	62.8%	0.101	14%

   5. Qualitative Analysis

      Figures 8, 9, and 10 show representative outputs across different spatial congrationsleft/right placement, surrounding arrangement, and vertical stacking. All are handled correctly with physically plausible positions.

      (a) NLP-Extracted Scene Layout (b) Generated 3D Meshes

      Fig. 8: Example 1: A big red cube to the left of a little blue sphere. The parser recognizes two objects with features (color: red/blue, shape: cube/sphere, size: big/little) and spatial relation (left).

      (a) NLP-Extracted Scene Layout (b) Generated 3D Meshes

      Fig. 9: Example 2: Three golden cylinders around a giant glass pyramid. The NLP parses four items: three cylinders (golden, cylinder, medium) surrounding one pyramid (glass, pyramid, giant).

      (a) NLP-Extracted Scene Layout (b) Generated 3D Meshes

      Fig. 10: Example 3: A small pink donut hanging above a large black cube. The parser recognizes two objects with different sizes and vertical spatial relation.

   6. Cross-Dataset Generalization

      Figure 11 shows the system generalizing reasonably well to real-world objects from Text2Shape and ShapeNet. Scenes with wooden chairs, lamps, mugs, laptops, books, and plants all get plausible layouts, suggesting the spatial and physical reasoning isnt tied to CLEVRs synthetic primitives.

      Fig. 11: Qualitative results on real-world objects from Text2Shape and ShapeNet. Top Left: A wooden chair near a round table. Top Right: A red lamp atop a white desk. Bottom Left: A black mug behind a silver laptop. Bottom Right: Two blue books next to a green plant.

   7. Error Analysis

      Table III breaks down performance by question type. Color queries are the weakest point (12.4%), driven by confusion between perceptually similar hues. Shape queries do better (43.0%) but still suffer from cylinder/cube confusion. Count queries (35.2%) and existence queries (52.1%) suffer from under-counting and false positives. Relation queries (28.7%) mainly fail on left/right disambiguation.

      TABLE III: Error Analysis by Question Type

      Question Type

      Accuracy

      Common Error

      Color queries	12.4%	Confusing similar colors
      Shape queries	43.0%	Cylinder vs. cube confusion
      Count queries	35.2%	Under/over counting
      Existence queries	52.1%	False positives
      Relation queries	28.7%	Left/right confusion

   8. Computational Efciency

   Excluding Shap-E, inference runs at roughly 0.25 seconds per query on an NVIDIA T4 GPU. Shap-E itself is the bottleneck30 90 seconds per object (about 45.3 seconds for diffusion, 12.1 seconds for decoding). This is the most pressing practical limitation.

7. DISCUSSION

   1. Theoretical Implications

      l l

      Several observations from our experiments have broader theoretical interest. Spatial Attention Bias: Relation accuracy jumps from 47.3% to 62.1% when we add spatial biasa substantial gain that strongly suggests geometric information isnt just complementary but essential for relational reasoning. Inspecting the learned bias reveals an approximate linear relationship ij 0.3 pi pj 2 + 0.5. The model has learned to down-weight attention between distant objects, a behavior that emerges from data rather than being hard-coded.

      Message Passing: The 11.3% improvement in Relation F1 attributable to the GNN conrms that global relational structure matters for multi-object scenes, not just local or pairwise reasoning.

      End-to-End Latent Alignment: Switching from pipeline to end-to-end LAM yields 4.1 VQA points and a 10% PVS improvement. Reasoning and generation genuinely benet from being trained together rather than in isolation.

      Fourier Features: The 23% drop in coordinate MSE when Fourier features are included is consistent with spectral bias theorythey help networks overcome their tendency to ignore high frequencies.

      Physical Losses: The 87.8% reduction in physics violations is striking. Each loss type contributes something distinctremoving support loss alone raises PVS to 22%, while removing gravity loss has a smaller but meaningful effect (14%). These constraints arent redundant.

   2. Comparison with Existing Work

      Table IV puts our results in context. On CLEVR VQA, our full model reaches 63.0%, below FiLM (67.1%), NS-VQA (68.5%), and MAC Network (69.2%). But those methods operate on rendered images and use visual features. Our system works exclusively from text and must simultaneously reason about 3D structure and physical plausibility. A 63.0% text-only result alongside an 87.8% PVS reduction is a different trade-off entirely.

      TABLE IV: Comparison with Existing Methods on CLEVR

      Method	VQA Accuracy
      CNN+LSTM Baseline	41.2%
      FiLM [41]	67.1%
      NS-VQA [24]	68.5%
      MAC Network [42]	69.2%
      Ours (w/o GNN/Physics)	48.2%
      Ours (full)	63.0%

   3. Limitations

      Lets be upfront about the limitations. First, training primarily on CLEVRs synthetic scenes limits real-world generalization. Second, Shap-E produces geometrically coherent but sometimes low-detail meshes. Third, slot attention caps scenes at 8 objects. Fourth, we only support 8 spatial relation typesbetween, surrounding, and aligned with arent modeled. Fifth, size is categorical rather than continuous. Sixth, 12.4% color accuracy is disappointingly low. Seventh, 3090 second per-object generation time makes real-time use infeasible.

   4. Future Work

   Several directions seem promising. Training on larger, more varied data combining CLEVR with Objaverse and ShapeNet would improve generalization. Richer GNN architectures could support continuous object counts and a wider relation vocabulary. Forward physics simulation during training could push physical consistency further. Shap-E latency could be reduced through latent caching, knowledge distillation, or more efcient sampling. Interactive scene editing from follow-up text is a natural extension. Contrastive training could improve color and shape discrimination. Multi-modal training combining text, images, and 3D data would broaden applicability.

8. CONCLUSION

Weve introduced a system for text-driven 3D scene generation that takes physical plausibility seriously from the start. It combines a multi-pass NLP parser, a Spatially Grounded Transformer, a Graph Neural Scene Reasoner, a Latent Alignment Module, and a Physics-Aware Scene Consistency Module into an end-to-end trainable architecture.

The results are encouraging. The full model achieves 63.0% VQA accuracy on CLEVR from text alone. The GNN brings relation F1 to 73.4%. Most strikingly, the physics module reduces physical violations by 87.8%, from 41% PVS down to 5%.

l l

LAM enables gradient ow from generation back into reasoning. The learned spatial bias follows ij 0.3 pi pj 2 + 0.5, revealing something interpretable about how attention organizes itself around geometry.

That said, CLEVRs synthetic data is a real constraint. The color prediction weakness (12.4%) and Shap-E generation latency are pressing practical issues. The 8-object ceiling and limited relation vocabulary need addressing before deployment in unconstrained scenes.

We hope this work contributes three things: the PVS metric for evaluating physical plausiility, physics-aware loss functions transferable to other scene generation tasks, and a demonstration that spatial grounding in transformer attention is practical and effective for multimodal reasoning. Code and trained models will allow researchers to build on these components in robotics simulation, autonomous driving, and interactive virtual environment design.

DECLARATIONS

Availability of supporting data

The data is available upon request.

Competing interests

The author declares no competing interests.

Funding

This research received no specic grant from any funding agency in the public, commercial, or not-for-prot sectors.

Authors contributions

Baibhab Das: Conceptualization, Methodology, Software, Validation, Formal Analysis, Investigation, Data Curation, Writing Original Draft, Writing Review & Editing, Visualization, Supervision, Project Administration.

Acknowledgements Not Applicable.

REFERENCES

1. A. Karpathy, L. Fei-Fei, Deep visual-semantic alignments for generating image descriptions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 31283137.

2. O. Vinyals, A. Toshev, S. Bengio, D. Erhan, Show and tell: A neural image caption generator, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 31563164.

3. D. Gopinath, J. Oh, Learning to ground 3D object affordances from natural language instructions, in: Conference on Robot Learning (CoRL), 2022, pp. 12341245.

4. M. Birsak, F. Rist, M. Wimmer, Procedural generation of 3D scenes from natural language descriptions, Computers & Graphics 92 (2020) 4555.

5. C. Paxton, Y. Bisk, J. Thomason, A. Byravan, D. Fox, Language to action: Learning to follow natural language instructions with a partially observable Markov decision process, in: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018, pp. 19.

6. D. Elliott, M. Frank, Multi-modal language processing: A survey, Foundations and Trends in Information Retrieval 12 (2) (2018) 89219.

7. A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, M. Chen, GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models, in: International Conference on Machine Learning (ICML), 2022, pp. 1678416804.

8. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, High-resolution image synthesis with latent diffusion models, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 1068410695.

9. H. Jun, A. Nichol, Shap-E: Generating conditional 3D implicit functions, arXiv preprint arXiv:2305.02463 (2023).

10. A. Nichol, H. Jun, P. Dhariwal, P. Mishkin, M. Chen, Point-E: A system for generating 3D point clouds from complex prompts, arXiv preprint arXiv:2212.08751 (2022).

11. S. Liu, X. Chen, Y. Guo, L. Zhang, B. Sheng, Generating 3D scenes from text: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (8) (2023) 95129532.

12. R. Hu, A. Rohrbach, T. Darrell, K. Saenko, Language-conditioned semantic segmentation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 584593.

13. J. Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. L. Zitnick, R. Girshick, CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 29012910.

14. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., Learning transferable visual models from natural language supervision, in: International Conference on Machine Learning (ICML), 2021, pp. 87488763.

15. M. Honnibal, I. Montani, spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing, To appear (2017).

16. M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, A. Farhadi, Objaverse: A universe of annotated 3D objects, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 1314213153.

17. A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al., ShapeNet: An information-rich 3D model repository, arXiv preprint arXiv:1512.03012 (2015).

18. K. Chen, C. B. Choy, M. Savva, A. X. Chang, T. Funkhouser, S. Savarese, Text2Shape: Generating shapes from natural language, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 876885.

19. J. Ho, A. Jain, P. Abbeel, Denoising diffusion probabilistic models, in: Advances in Neural Information Processing Systems (NeurIPS), 2020, pp. 68406851.

20. J. Ho, T. Salimans, Classier-free diffusion guidance, arXiv preprint arXiv:2207.12598 (2022).

21. J. Wu, C. Zhang, T. Xue, B. Freeman, J. Tenenbaum, Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling, in: Advances in Neural Information Processing Systems (NeurIPS), 2016, pp. 8290.

22. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, R. Ng, NeRF: Representing scenes as neural radiance elds for view synthesis, in: European Conference on Computer Vision (ECCV), 2020, pp. 405421.

23. D. Ritchie, K. Wang, Y. Lin, Fast and exible indoor scene synthesis via deep convolutional generative models, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 61826190.

24. J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, J. Wu, The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision, in: International Conference on Learning Representations (ICLR), 2019.

25. S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, M. Shah, Transformers in vision: A survey, ACM Computing Surveys 54 (10s) (2022) 141.

26. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., An image is worth 16×16 words: Transformers for image recognition at scale, in: International Conference on Learning Representations (ICLR), 2021.

27. N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko, End-to-end object detection with transformers, in: European Conference on Computer Vision (ECCV), 2020, pp. 213229.

28. F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, T. Kipf, Object-centric learning with slot attention, in: Advances in Neural Information Processing Systems (NeurIPS), 2020, pp. 1152511538.

29. I. Mani, B. Pustejovsky, Temporal and spatial reasoning in natural language, Computational Linguistics 34 (4) (2008) 635641.

30. J. R. Hobbs, Spatial reasoning in natural language, Computational Linguistics 20 (2) (1994) 307313.

31. A. Kordjamshidi, M. Van Otterlo, M. Moens, Spatial role labeling: Towards extraction of spatialrelations from natural language, ACM Transactions on Speech and Language Processing 8 (3) (2011) 136.

32. R. Li, A. L. S. Ma, S. F. Su, A survey of spatial language understanding in natural language processing, Natural Language Engineering 27 (6) (2021) 697725.

33. Y. Lu, R. Li, B. Yang, S. Gould, D. Hoiem, Spatial reasoning with transformers for visual question answering, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 15681577.

34. T. Chen, S. Saxena, L. Li, D. J. Fleet, G. Hinton, A simple framework for contrastive learning of visual representations, in: International Conference on Machine Learning (ICML), 2020, pp. 15971607.

35. N. Kitaev, L. Kaiser, A. Levskaya, Reformer: The efcient transformer, in: International Conference on Learning Representations (ICLR), 2020.

36. P. Battaglia, R. Pascanu, M. Lai, D. J. Rezende, K. Kavukcuoglu, Interaction networks for learning about objects, relations and physics, in: Advances in Neural Information Processing Systems (NeurIPS), 2016, pp. 45024510.

37. Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, A. Torralba, Learning to simulate complex physics with graph networks, in: International Conference on Machine Learning (ICML), 2019, pp. 38783887.

38. G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, L. Yang, Physics-informed machine learning, Nature Reviews Physics 3 (2021) 422440.

39. V. Sanh, L. Debut, J. Chaumond, T. Wolf, DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, arXiv preprint arXiv:1910.01108 (2019).

40. M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron, R. Ng, Fourier features let networks learn high frequency functions in low dimensional domains, in: Advances in Neural Information Processing Systems (NeurIPS), 2020, pp. 75377547.

41. E. Perez, F. Strub, H. de Vries, V. Dumoulin, A. Courville, FiLM: Visual reasoning with a general conditioning layer, in: AAAI Conference on Articial Intelligence, 2018, pp. 39423951.

42. D. A. Hudson, C. D. Manning, Compositional attention networks for machine reasoning, in: International Conference on Learning Representations (ICLR), 2018.