Add initial QVM CLI, Nix flake, scripts and README
This commit is contained in:
Joshua Bell
parent
25b1cca0e6
commit
8534f7efb9
14 changed files with 2359 additions and 0 deletions
549
README.md
Normal file
549
README.md
Normal file
|
|
@ -0,0 +1,549 @@
|
|||
# QVM - Lightweight QEMU Development VM Wrapper
|
||||
|
||||
A standalone CLI tool for running commands in an isolated NixOS VM with persistent state and shared caches.
|
||||
|
||||
## Motivation
|
||||
|
||||
Running AI coding agents in isolation presents a security challenge. Containers provide some isolation, but kernel exploits remain a real attack surface. QVM uses full VM isolation for stronger security guarantees while maintaining developer ergonomics.
|
||||
|
||||
**Why QVM?**
|
||||
|
||||
- **VM isolation over container isolation** - Hypervisor boundary is fundamentally stronger than kernel namespaces
|
||||
- **One master image, shared caches** - Single ~7GB base image instead of per-project images
|
||||
- **Transparent workspace mounting** - Current directory automatically available in VM
|
||||
- **Persistent state** - VM overlay preserves installed tools and configuration
|
||||
- **Shared build caches** - Cargo, pnpm, and sccache caches shared across all projects
|
||||
|
||||
**Primary use case:** Running AI coding agents (opencode, aider, cursor) in isolation to prevent unintended host filesystem access while maintaining build cache performance.
|
||||
|
||||
## Installation
|
||||
|
||||
### System-wide installation (NixOS flake)
|
||||
|
||||
Add QVM to your NixOS configuration:
|
||||
|
||||
```nix
|
||||
{
|
||||
inputs.qvm.url = "github:yourusername/qvm";
|
||||
|
||||
environment.systemPackages = [
|
||||
inputs.qvm.packages.${system}.default
|
||||
];
|
||||
}
|
||||
```
|
||||
|
||||
### Direct execution
|
||||
|
||||
Run without installation:
|
||||
|
||||
```bash
|
||||
nix run github:yourusername/qvm -- start
|
||||
```
|
||||
|
||||
### Development shell
|
||||
|
||||
For local development:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/yourusername/qvm
|
||||
cd qvm
|
||||
nix develop
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Start the VM (auto-builds base image on first run)
|
||||
qvm start
|
||||
|
||||
# Run commands in VM with current directory mounted
|
||||
qvm run opencode
|
||||
qvm run cargo build --release
|
||||
qvm run npm install
|
||||
|
||||
# Interactive SSH session
|
||||
qvm ssh
|
||||
|
||||
# Check VM status
|
||||
qvm status
|
||||
|
||||
# Stop VM
|
||||
qvm stop
|
||||
```
|
||||
|
||||
The first `qvm start` will build the base image, which takes several minutes. Subsequent starts are fast (3-5 seconds).
|
||||
|
||||
## Commands
|
||||
|
||||
### qvm start
|
||||
|
||||
Start the VM daemon. Creates base image and overlay if they don't exist.
|
||||
|
||||
```bash
|
||||
qvm start
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
- Checks if VM already running (no-op if yes)
|
||||
- Builds base image if missing (via `qvm rebuild`)
|
||||
- Creates overlay.qcow2 if missing (copy-on-write backed by base image)
|
||||
- Launches QEMU with KVM acceleration
|
||||
- Mounts all registered workspaces and cache directories via 9p
|
||||
- Waits for SSH to become available
|
||||
- Exits when SSH is ready (VM runs as background daemon)
|
||||
|
||||
### qvm stop
|
||||
|
||||
Gracefully stop the running VM.
|
||||
|
||||
```bash
|
||||
qvm stop
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
- Sends ACPI shutdown signal to VM
|
||||
- Waits up to 30 seconds for graceful shutdown
|
||||
- Force kills QEMU if timeout exceeded
|
||||
- Cleans up PID file
|
||||
- No-op if VM not running
|
||||
|
||||
### qvm run
|
||||
|
||||
Execute a command in the VM with current directory mounted as workspace.
|
||||
|
||||
```bash
|
||||
qvm run <command> [args...]
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
qvm run cargo build
|
||||
qvm run npm install
|
||||
qvm run "ls -la"
|
||||
qvm run bash # Interactive shell in VM
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
1. Generates hash from current directory absolute path
|
||||
2. Registers workspace in `~/.local/state/qvm/workspaces.json`
|
||||
3. Auto-starts VM if not running
|
||||
4. Checks if workspace is mounted (warns to restart VM if newly registered)
|
||||
5. SSHs into VM and executes: `cd /workspace/{hash} && <command>`
|
||||
6. Streams stdout/stderr to terminal
|
||||
7. Exits with command's exit code
|
||||
8. VM stays running for next command
|
||||
|
||||
**Note:** Workspaces are mounted at VM startup. If you run from a new directory, you'll need to restart the VM to mount it.
|
||||
|
||||
### qvm ssh
|
||||
|
||||
Open interactive SSH session or run command in VM.
|
||||
|
||||
```bash
|
||||
qvm ssh # Interactive shell
|
||||
qvm ssh -c "command" # Run single command
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
qvm ssh # Drop into root shell
|
||||
qvm ssh -c "systemctl status" # Check systemd status
|
||||
qvm ssh -c "df -h" # Check disk usage
|
||||
```
|
||||
|
||||
Connects as root user. Password: `root` (if needed, though SSH key auth is configured).
|
||||
|
||||
### qvm status
|
||||
|
||||
Show VM state, SSH port, and mounted workspaces.
|
||||
|
||||
```bash
|
||||
qvm status
|
||||
```
|
||||
|
||||
**Example output:**
|
||||
```
|
||||
VM Status: Running
|
||||
PID: 12345
|
||||
SSH Port: 2222
|
||||
|
||||
Mounted Workspaces:
|
||||
abc12345 -> /home/josh/projects/qvm
|
||||
def67890 -> /home/josh/projects/myapp
|
||||
|
||||
Cache Directories:
|
||||
/cache/cargo -> ~/.cache/qvm/cargo-home
|
||||
/cache/target -> ~/.cache/qvm/cargo-target
|
||||
/cache/pnpm -> ~/.cache/qvm/pnpm-store
|
||||
/cache/sccache -> ~/.cache/qvm/sccache
|
||||
```
|
||||
|
||||
### qvm rebuild
|
||||
|
||||
Rebuild the base VM image from NixOS flake.
|
||||
|
||||
```bash
|
||||
qvm rebuild
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
- Runs `nix build` on `~/.config/qvm/flake`
|
||||
- Copies result to `~/.local/share/qvm/base.qcow2`
|
||||
- Warns if VM is running (restart required to use new image)
|
||||
|
||||
**Use when:**
|
||||
- Customizing VM configuration (edited `~/.config/qvm/flake/flake.nix`)
|
||||
- Updating base image with new packages or settings
|
||||
- Pulling latest changes from upstream NixOS
|
||||
|
||||
### qvm reset
|
||||
|
||||
Delete overlay and start fresh. Keeps base image intact.
|
||||
|
||||
```bash
|
||||
qvm reset
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
- Stops VM if running
|
||||
- Deletes `overlay.qcow2` (all VM state changes)
|
||||
- Deletes `workspaces.json` (registered workspaces)
|
||||
- Next `qvm start` creates fresh overlay from base image
|
||||
|
||||
**Use when:**
|
||||
- VM state is corrupted
|
||||
- Want to return to clean base image state
|
||||
- Testing fresh install scenarios
|
||||
|
||||
## Directory Layout
|
||||
|
||||
QVM uses XDG-compliant directories:
|
||||
|
||||
```
|
||||
~/.config/qvm/
|
||||
└── flake/ # User's customizable NixOS flake
|
||||
├── flake.nix # VM system configuration
|
||||
└── flake.lock # Pinned dependencies
|
||||
|
||||
~/.local/share/qvm/
|
||||
└── base.qcow2 # Base VM image (~7GB, read-only)
|
||||
|
||||
~/.local/state/qvm/
|
||||
├── overlay.qcow2 # Persistent VM state (copy-on-write)
|
||||
├── vm.pid # QEMU process ID
|
||||
├── ssh.port # SSH forwarded port
|
||||
├── serial.log # VM console output
|
||||
└── workspaces.json # Registered workspace mounts
|
||||
|
||||
~/.cache/qvm/
|
||||
├── cargo-home/ # Shared cargo registry/cache
|
||||
├── cargo-target/ # Shared cargo build artifacts
|
||||
├── pnpm-store/ # Shared pnpm content-addressable store
|
||||
└── sccache/ # Shared compilation cache
|
||||
```
|
||||
|
||||
### Inside the VM
|
||||
|
||||
```
|
||||
/workspace/
|
||||
├── abc12345/ # Mounted from /home/josh/projects/qvm
|
||||
└── def67890/ # Mounted from /home/josh/projects/myapp
|
||||
|
||||
/cache/
|
||||
├── cargo/ # Mounted from ~/.cache/qvm/cargo-home
|
||||
├── target/ # Mounted from ~/.cache/qvm/cargo-target
|
||||
├── pnpm/ # Mounted from ~/.cache/qvm/pnpm-store
|
||||
└── sccache/ # Mounted from ~/.cache/qvm/sccache
|
||||
```
|
||||
|
||||
Environment variables in VM:
|
||||
- `CARGO_HOME=/cache/cargo`
|
||||
- `CARGO_TARGET_DIR=/cache/target`
|
||||
- `PNPM_HOME=/cache/pnpm`
|
||||
- `SCCACHE_DIR=/cache/sccache`
|
||||
|
||||
## Workspace Management
|
||||
|
||||
When you run `qvm run` from different directories, each gets registered and mounted:
|
||||
|
||||
```bash
|
||||
cd ~/projects/myapp
|
||||
qvm run cargo build # Workspace abc12345
|
||||
|
||||
cd ~/projects/other
|
||||
qvm run npm install # Workspace def67890
|
||||
```
|
||||
|
||||
Both workspaces are mounted simultaneously in the VM. The hash is derived from the absolute path, so the same directory always maps to the same workspace ID.
|
||||
|
||||
**Important:** Workspaces are mounted at VM start time. If you run from a new directory:
|
||||
1. Workspace gets registered in `workspaces.json`
|
||||
2. `qvm run` will detect it's not mounted and warn you
|
||||
3. Restart VM to mount: `qvm stop && qvm start`
|
||||
|
||||
## Cache Sharing
|
||||
|
||||
Build caches are shared between host and VM via 9p mounts:
|
||||
|
||||
**Cargo:**
|
||||
- Registry and crate cache shared across all projects
|
||||
- Each project still uses its own `Cargo.lock`
|
||||
- Different versions coexist peacefully
|
||||
|
||||
**pnpm:**
|
||||
- Content-addressable store shared
|
||||
- Each project links to shared store
|
||||
- Massive disk space savings
|
||||
|
||||
**sccache:**
|
||||
- Compilation cache shared
|
||||
- Speeds up repeated builds across projects
|
||||
|
||||
All caches persist across VM restarts and resets (they live in `~/.cache/qvm`, not in the overlay).
|
||||
|
||||
## Customization
|
||||
|
||||
### Editing VM Configuration
|
||||
|
||||
The VM is defined by a NixOS flake at `~/.config/qvm/flake/flake.nix`. Edit this file to customize the VM.
|
||||
|
||||
**Default packages included:**
|
||||
- git, vim, tmux, htop
|
||||
- curl, wget, jq, ripgrep, fd
|
||||
- opencode (AI coding agent)
|
||||
- Language toolchains can be added
|
||||
|
||||
**Example: Add Rust and Node.js:**
|
||||
|
||||
```nix
|
||||
environment.systemPackages = with pkgs; [
|
||||
# ... existing packages ...
|
||||
|
||||
# Add language toolchains
|
||||
rustc
|
||||
cargo
|
||||
nodejs_22
|
||||
python3
|
||||
];
|
||||
```
|
||||
|
||||
**Example: Add custom shell aliases:**
|
||||
|
||||
```nix
|
||||
environment.shellAliases = {
|
||||
"ll" = "ls -lah";
|
||||
"gst" = "git status";
|
||||
};
|
||||
```
|
||||
|
||||
**Example: Include your dotfiles:**
|
||||
|
||||
```nix
|
||||
environment.etc."vimrc".source = /path/to/your/vimrc;
|
||||
```
|
||||
|
||||
**Apply changes:**
|
||||
|
||||
```bash
|
||||
# Edit the flake
|
||||
vim ~/.config/qvm/flake/flake.nix
|
||||
|
||||
# Rebuild base image
|
||||
qvm rebuild
|
||||
|
||||
# Restart VM to use new image
|
||||
qvm stop
|
||||
qvm start
|
||||
```
|
||||
|
||||
### Resource Allocation
|
||||
|
||||
Default resources:
|
||||
- **Memory:** 8GB
|
||||
- **CPUs:** 4 cores
|
||||
- **Disk:** 20GB
|
||||
|
||||
To customize, set environment variables before `qvm start`:
|
||||
|
||||
```bash
|
||||
export QVM_MEMORY="16G"
|
||||
export QVM_CPUS="8"
|
||||
qvm start
|
||||
```
|
||||
|
||||
These will be configurable in a config file in future versions.
|
||||
|
||||
## Security Model
|
||||
|
||||
### VM Isolation Benefits
|
||||
|
||||
**Why VM over containers?**
|
||||
- Container escapes via kernel exploits are well-documented
|
||||
- VM escapes require hypervisor exploits (far rarer)
|
||||
- For long-running unattended AI sessions, VM isolation provides stronger guarantees
|
||||
|
||||
### 9p Mount Security
|
||||
|
||||
Filesystem sharing uses `security_model=mapped-xattr`:
|
||||
- No direct passthrough to host filesystem
|
||||
- Only explicitly mounted directories are visible
|
||||
- Host filesystem outside mounts is completely invisible to VM
|
||||
- File ownership and permissions mapped via extended attributes
|
||||
|
||||
**What the VM can access:**
|
||||
- Registered workspaces (directories you ran `qvm run` from)
|
||||
- Shared cache directories
|
||||
- Nothing else on the host
|
||||
|
||||
**What the VM cannot access:**
|
||||
- Your home directory (except mounted workspaces)
|
||||
- System directories
|
||||
- Other users' files
|
||||
- Any unmounted host paths
|
||||
|
||||
### Network Isolation
|
||||
|
||||
VM uses QEMU user-mode networking:
|
||||
- VM can make outbound connections
|
||||
- No inbound connections to VM except forwarded SSH port
|
||||
- SSH port forwarded to random high port on localhost only
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### VM won't start
|
||||
|
||||
Check if QEMU/KVM is available:
|
||||
|
||||
```bash
|
||||
qemu-system-x86_64 --version
|
||||
lsmod | grep kvm
|
||||
```
|
||||
|
||||
Check the serial log for errors:
|
||||
|
||||
```bash
|
||||
cat ~/.local/state/qvm/serial.log
|
||||
```
|
||||
|
||||
### SSH timeout
|
||||
|
||||
If SSH doesn't become ready within 60 seconds:
|
||||
|
||||
1. Check if VM process is running: `ps aux | grep qemu`
|
||||
2. Check serial log: `cat ~/.local/state/qvm/serial.log`
|
||||
3. Try increasing timeout (future feature)
|
||||
|
||||
### Workspace not mounted
|
||||
|
||||
If you see "Workspace not mounted in VM":
|
||||
|
||||
```bash
|
||||
qvm stop
|
||||
qvm start
|
||||
qvm run <command>
|
||||
```
|
||||
|
||||
Workspaces must be registered before VM start.
|
||||
|
||||
### Build cache not working
|
||||
|
||||
Verify cache directories exist and are mounted:
|
||||
|
||||
```bash
|
||||
qvm ssh -c "ls -la /cache"
|
||||
qvm ssh -c "echo \$CARGO_HOME"
|
||||
```
|
||||
|
||||
Check that cache directories on host exist:
|
||||
|
||||
```bash
|
||||
ls -la ~/.cache/qvm/
|
||||
```
|
||||
|
||||
### VM is slow
|
||||
|
||||
Ensure KVM is enabled (not using emulation):
|
||||
|
||||
```bash
|
||||
lsmod | grep kvm_intel # or kvm_amd
|
||||
```
|
||||
|
||||
Check resource allocation:
|
||||
|
||||
```bash
|
||||
qvm status # Shows allocated CPUs/memory (future feature)
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
**Explicit exclusions:**
|
||||
- **Multi-VM:** Only one VM at a time
|
||||
- **Per-project configs:** Single global VM (use other tools for project-specific VMs)
|
||||
- **Cross-platform:** Linux + KVM only (no macOS/Windows)
|
||||
- **GUI:** Headless only, no desktop environment
|
||||
- **Snapshots:** Only overlay reset, no checkpoint/restore
|
||||
|
||||
## Dependencies
|
||||
|
||||
Required on host system:
|
||||
- `qemu` (with KVM support)
|
||||
- `nix` (for building VM images)
|
||||
- `openssh` (SSH client)
|
||||
- `jq` (JSON processing)
|
||||
- `nc` (netcat, for port checking)
|
||||
|
||||
All dependencies are included automatically when installing via Nix flake.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
HOST VM
|
||||
──────────────────────────────── ──────────────────────────
|
||||
|
||||
~/.local/share/qvm/base.qcow2 → (read-only base image)
|
||||
↓
|
||||
~/.local/state/qvm/overlay.qcow2 → (persistent changes)
|
||||
|
||||
|
||||
~/.cache/qvm/cargo-home/ ──9p──→ /cache/cargo/
|
||||
~/.cache/qvm/cargo-target/ ──9p──→ /cache/target/
|
||||
~/.cache/qvm/pnpm-store/ ──9p──→ /cache/pnpm/
|
||||
~/.cache/qvm/sccache/ ──9p──→ /cache/sccache/
|
||||
|
||||
$(pwd) ──9p──→ /workspace/{hash}/
|
||||
```
|
||||
|
||||
**Image layering:**
|
||||
- Base image contains NixOS system from flake
|
||||
- Overlay is copy-on-write layer for runtime changes
|
||||
- `qvm reset` deletes overlay, preserves base
|
||||
- `qvm rebuild` updates base, keeps overlay
|
||||
|
||||
**9p virtfs mounts:**
|
||||
- Used for workspace and cache sharing
|
||||
- `security_model=mapped-xattr` for security
|
||||
- `msize=104857600` for performance (100MB transfer size)
|
||||
- Mounts configured at VM start, no hotplug
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions welcome! This is a simple Bash-based tool designed to be readable and hackable.
|
||||
|
||||
**Key files:**
|
||||
- `bin/qvm` - Main dispatcher
|
||||
- `bin/qvm-*` - Subcommand implementations
|
||||
- `lib/common.sh` - Shared utilities and paths
|
||||
- `flake/default-vm/flake.nix` - Default VM template
|
||||
|
||||
**Development:**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/yourusername/qvm
|
||||
cd qvm
|
||||
nix develop
|
||||
./bin/qvm start
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
Loading…
Add table
Add a link
Reference in a new issue