How to control the novel_view?

[PlinkPlunkT]

Hi,
Thanks for your amazing work!

I successfully run the model and do inference things. However, I've encountered a problem. I'm using the perspective of a forward-facing camera as input to generate a new perspective video that turns 30 degrees to the left, but the new video I generate always turns into an in-car perspective. I've tried many prompts, but none of them have generated the correct 30-degree left-facing perspective. Can you tell me how to achieve this?

What I input is a forward facing video just like video in your example:

then I use pan_left_30_fixed.json：

  {
    "name": "pan_left_30_fixed",
    "description": "Fixed pan left 30 degrees from frame 0",
    "mode": "relative",
    "num_frames": 81,
    "trajectory": {
      "frame_indices": [0, 80],
      "frame_matrices": [
        [
          [0.8660254, 0.0, -0.5, 0.0],
          [0.0,       1.0,  0.0, 0.0],
          [0.5,       0.0,  0.8660254, 0.0],
          [0.0,       0.0,  0.0, 1.0]
        ],
        [
          [0.8660254, 0.0, -0.5, 0.0],
          [0.0,       1.0,  0.0, 0.0],
          [0.5,       0.0,  0.8660254, 0.0],
          [0.0,       0.0,  0.0, 1.0]
        ]
      ]
    }
  }

and use command:

python inference.py \
      --input_path /path/to/front_camera/ \
      --trajectory_file pan_left_30_fixed.json \
      --prompt "A driving video recorded from an exterior camera mounted on a vehicle, showing outdoor street scene with roads, buildings and other vehicles. No interior car cabin
  visible." \
      --output_path outputs/pan_left_30_fixed.mp4

But I got a new video the camera is in the car, and I can see the steering wheel even the driver.

I'd like to know how you generate the left and right views from the forward-facing camera. Could you please tell me?

Thank you very much.

[YuxueYang1204]

@PlinkPlunkT Sorry for the late reply! The frame_matrices are incremental — each matrix is applied on top of the previous one. So if you set both timestamp 0 and timestamp 80 to the same pan-left-30° transformation matrix, the camera will pan 30° at frame 0, and then pan another 30° by frame 80 (60° total).

If your intention is to only pan left 30° at the beginning and hold that pose, you should set:

• Frame 0 → pan-left-30° matrix
• Frame 80 → identity matrix (no additional transformation)

Hope that helps!