One framework, multiple policies
Connect GPT vision and structured actions, LIBERO-specific local VLA presets, or Jev text primitives. Each policy keeps its declared inputs and action protocol.
Explore the architectureOPEN ROBOT MANIPULATION
See.
Act.
Observe again.
Connect model decisions to a physical feedback loop. Explore MuJoCo, LIBERO, robosuite and RoboCasa through one local workspace, with explicit actions and inspectable feedback.
Inspect a saved LIBERO run in your browser. The live, modular workspace runs on your machine.
Python 3.12MuJoCoLIBEROrobosuiteRoboCasa
RECORDED POLICY LOOPGPT · LIBERO · PANDA
0.00 / — s
Simulation time · Model waits omittedFull episode
MODEL ACTION NOTE · RECORDED—
- Requested action
- —
- Previous action feedback
- —
Feedback to the model uses only camera images and proprioception available on a real robot; no privileged simulator ground truth is provided. The controller computes TCP error from the model's previous command and the measured post-action TCP position, without object ground truth. Action execution completion does not mean task success.
Pause to read the model's saved action note. This page makes no model requests.
∥
The policy receives only declared sensor observations.
The evaluator stays separate.
RECORDED SKILLS
See a task through.
Select a task to play its recorded actions above. Each card is one independently verified episode, not a benchmark success rate.
Run time includes model planning and waiting, simulation execution and recording; environment initialization and policy loading are excluded. Video playback follows simulation time.
01 / THE FRAMEWORK
Close the loop. Inspect every step.
Connect environment adapters, model policies and local control. Inspect the requested action, the measured motion and the independently recorded result.
01
Observe
Camera RGB where supported Robot proprioception
SENSOR INPUT02
Decide
GPT / VLA policies Jev text primitives
POLICY03
Act
Validate and track targets MuJoCo backends
LOCAL CONTROL04
Check
New images and motion Execution status and error
FEEDBACKFresh sensor evidence for the next decision
Independent evaluationObject ground truth, rewards and success signals go to the viewer and result records only.EVALUATOR ≠ POLICY INPUT
Control a 6D TCP target
Jog XYZ, rotate Rx/Ry/Rz, or combine translation and rotation in fixed or tool axes. Inspect position and orientation error. Manual control needs no model key.
Understand manual pose controlKeep actions and evidence together
Inspect decision-level observations, action notes and feedback across backends. LIBERO additionally supports complete control-step recordings and continuous MP4 export.
Record and exportFour environments. Explicit boundaries.
Choose the scene in the local workspace. Optional backends keep separate runtimes, controllers and independent task evaluators.
Built-in MuJoCo
Panda and ARX5 tabletop scenes for manual control, pick-and-place and pushing experiments.
Setup and scopeLIBERO
Official task suites and initializations. The recorded GPT examples and local VLA presets on this page use LIBERO.
Setup and scoperobosuite
Five Panda tasks: Lift, Stack, PickPlaceCan, Door and NutAssemblySquare, with dual-camera observation and local control.
Setup and scopeRoboCasa
Four kitchen tasks: open or close a drawer, open a cabinet, and position a mug. Arm and gripper control; no navigation planner.
Setup and scopeBackend integration is not a model success result. The existing LeRobot presets remain LIBERO-specific; robosuite and RoboCasa do not yet provide complete control-step video export.
THE LOCAL APPLICATION
A workspace for the whole loop.
Manual control, model runs and scene settings have dedicated panels beside the cameras. These actual local-application screenshots use the built-in Panda tabletop scene; the website does not run their controls.
01 / MANUAL CONTROL
02 / MODEL RUNS
Apply a scene when you are ready
Choose a backend and task, then apply to rebuild the environment. Editing a task instruction does not change the official evaluator. The stop control remains available across panels.
02 / THE EPISODE
One task. Every step recorded.
Actual simulator footage with an independent task result. This page plays saved media; it does not call a model or run a simulator.
RECORDED DEMO
—Loading the task record
Model and settings are loaded from the episode
COMPLETE EPISODE
Selected decision frames
Observation points from the same episode, paired with action notes. These are not continuous video.
01 / EXTERNALRGB 512
02 / WRISTRGB 512
— / —
Selected decision frames · Original order · Not continuous video
—model decisions
—native control steps
—ssimulation time
PendingIndependent task result
Loading run conditions
Task completion is determined by the independent simulator evaluator. One episode does not establish a benchmark success rate.
03 / MODEL CONNECTIONS
Supported models and configurations.
Connect an account-backed GPT model, a compatible vision API, a fixed local LeRobot preset, or Jev text primitives. Each route has its own authentication, observations and operating scope.
ACCOUNT / CODEX CLI
GPT via Codex
- Connection & configuration
- Use a recent official Codex CLI, sign in with your ChatGPT account, and choose a model ID available to that account. ManiLoop delegates authentication to the CLI.
- Policy inputs
- Task instruction, camera RGB, robot proprioception and permitted action feedback.
- Operating scope
- Structured TCP actions in an installed simulation backend. Run text, image and action-format diagnostics for the selected model before controlling the robot.
HOSTED / SELF-HOSTED
Compatible vision API
- Connection & configuration
- Set the service base URL, model ID and API key. The endpoint must support Responses, image input and JSON Schema structured output.
- Policy inputs
- Task instruction, camera RGB, robot proprioception and permitted action feedback through the Responses adapter.
- Operating scope
- Hosted or self-hosted vision services implementing that protocol. Select an available model and verify all three capabilities in the local diagnostics.
LOCAL / LANGUAGE-CONDITIONED
SmolVLA
- Connection & configuration
- Install or reuse the smolvla-libero preset and its pinned checkpoint in the separate .venv-vla inference environment.
- Policy inputs
- Language instruction, two 256 × 256 RGB views and the 8D robot proprioceptive state.
- Operating scope
- LIBERO Panda tasks with the preset observation processor and action protocol. Use the scene’s original English task instruction.
LOCAL / VISUAL IMITATION
ACT
- Connection & configuration
- Install or reuse the act-libero preset and its pinned community checkpoint in .venv-vla.
- Policy inputs
- Two 256 × 256 RGB views and the 8D robot proprioceptive state. This checkpoint does not consume language.
- Operating scope
- LIBERO only, with the checkpoint’s saved preprocessing and action chunks. Editing the text instruction does not change its task conditioning.
LOCAL / VISUAL DIFFUSION
Diffusion Policy
- Connection & configuration
- Install or reuse the diffusion-libero preset and its pinned community checkpoint in .venv-vla.
- Policy inputs
- Two 256 × 256 RGB views and the 8D robot proprioceptive state, updated each control step. This checkpoint does not consume language.
- Operating scope
- LIBERO only, preserving the checkpoint’s observation history, action chunks and sampling configuration.
TYPESAFE / TEXT PRIMITIVES
Jev
- Connection & configuration
- Provide a separate TypeSafe API key in the local workspace or TYPESAFE_API_KEY. No Jev weights or TypeSafe SDK installation is required.
- Policy inputs
- Explicit text instructions and public robot proprioception. Images are not sent to Jev.
- Operating scope
- Selects one bounded motion primitive per instruction in controlled timing. Local code handles units, coordinates and execution; this mode does not locate objects visually.
Local preset requirements
SmolVLA, ACT and Diffusion Policy share the isolated .venv-vla runtime and support CPU, Apple MPS or NVIDIA CUDA as available. Install the LIBERO backend separately. The adapter accepts these three catalog presets, not arbitrary checkpoint imports.
04 / GET STARTED
Run the loop on your machine.
Start with the local workspace and manual control, then choose an optional backend and policy. Simulation runtimes and VLA inference stay isolated.
01
Set up the main environment
macOS / Linux · Python 3.12
git clone https://github.com/Daniel-rmc/ManiLoop.git
cd ManiLoop
python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
.venv/bin/python -m maniloop demoOpen 127.0.0.1:8765 for the built-in tabletop workspace. Manual controls need no model key. Use another terminal, or stop with Ctrl+C, before installing an optional backend.
Full setup and Windows instructions02
Add LIBERO
A separate Python 3.10 simulation environment
.venv/bin/python -m pip install uv
.venv/bin/python scripts/setup_libero.py \
--uv .venv/bin/uv
.venv/bin/python -m maniloop demo \
--backend libero --port 8767Open 127.0.0.1:8767 on your machine. Git, network access and a supported graphics setup are required. VLA weights are optional.
Requirements and headless setupAdd another environment
Use the dedicated installers for robosuite and RoboCasa. Each creates its own Python 3.11 runtime; RoboCasa additionally needs kitchen assets.
CONNECT A MODEL
Give the policy a turn.
Choose GPT through a compatible image and structured-output API or official Codex login, a local LIBERO VLA preset, or Jev text primitives with a separate TypeSafe API key. Diagnose the connection before a small action.
Jev receives explicit instructions and public proprioception, not images. It selects a bounded primitive; local code handles units, coordinates and motion. It does not visually locate objects or provide autonomous grasping.
Model requests use the selected provider or account quota. Switching workspace panels does not trigger inference.
Connect your model. Inspect its actions.
Questions and contributions