Skip to content

Instantly share code, notes, and snippets.

@vyeevani

vyeevani/train-trajectory.ipynb

Created May 14, 2024 03:30
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save vyeevani/2b1509c3e610b6319a4fb4e4137cc9e1 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save vyeevani/2b1509c3e610b6319a4fb4e4137cc9e1 to your computer and use it in GitHub Desktop.
Download ZIP
diffusion learning with autoregressive perceivers
Display the source blob
Display the rendered blob
Raw
train-trajectory.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@vyeevani
Copy link
Author

vyeevani commented May 23, 2024

I wanted to get down my view on why this is needed:
desired_poses = input_project(einops.pack([self.desired_poses_start, desired_poses], "* r d")[0][:-1], self.desired_poses_positional_embedder)
The basic premise is that the future actions can view the denoised prior actions while making their own decision. This allows for autoregressive action prediction while we are still doing diffusion. Generalizing the perceiver has been challenging because I'm not sure how to properly abstract this without leaking details of how diffusion + autoregressive generation leaks into this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment