What Virtual Lab is
Virtual Lab gives university students virtual environments for their coursework, running both on our own machines and in the cloud, with SkyPilot and Kubernetes doing the scheduling. It has two services: the Laboratory Service, which spins up an environment when a student starts a lab, and Virtual Lab Security, which helps admins watch for threats across the university's infrastructure, on-premise and in the cloud.
Our lecturer gave the project to our team of three to get it ready for use in our department. It's still in development, so no students use it yet.
My two teammates built the screens below and the side that starts labs, including warm pods and session management. My work sits underneath.
The whole stack
- Web portal
- Vite, React 19, FastAPI, PostgreSQL
- Compute
- Kubernetes (K3s), Amazon EC2, SkyPilot
- Networking
- Tailscale, Cloudflare Tunnel
- Images & storage
- Amazon S3, Cloudflare R2, Docker Registry, Amazon ECR
- Security
- Trivy, Skopeo, Grafana, VictoriaLogs, Falco
- Infrastructure as code
- Terraform, Ansible
- CI/CD
- GitHub Actions
- Testing
- pytest
What I built
I joined my department's lab as an undergraduate researcher and handle DevOps for the team, mostly CI/CD. These are the parts I own.
- main focus
Image scan pipeline
From an instructor's upload to an approved image in ECR, end to end.
- security
One way into the registry
Separate push and pull sides, so only scanned images get in.
- access
Zero Trust access
Cloudflare Tunnel on K3s, set up with Ansible, so the cluster opens no ports to the internet.
- network
Tailscale ACLs
Who can talk to whom, written as code with tests.
- delivery
CI/CD
GitHub Actions pipelines for the platform. Argo CD is what I'm building next.
Why images needed a gate
Instructors bring their own container images for each lab, and those images end up running on machines students use. Before this, nothing checked them on the way in, and the registry could be written to from places that shouldn't be able to. That was the biggest hole in the system.
So the goal was simple to say: the only way an image gets into our registry is by passing a scan.
From upload to ECR
The instructor uploads straight to Cloudflare R2 with a one-time URL, so big files never pass through our backend. R2 fires an event, a Worker tells the backend, and the backend just marks the image as queued and answers.
Every 15 seconds a reconciler looks at what's queued and starts a Kubernetes Job for it. The Job runs four containers in order, all from one locked-down image I built:
- Fetch
- Downloads the uploaded file and checks its size and checksum against what storage recorded.
- Validate
- Checks the archive format and CPU architecture, without unpacking it.
- Scan
- Runs Trivy. A fixable critical vulnerability fails the image.
- Publish
- The only step that holds the registry password. Pushes the image and reports its digest.
If it passes, the image is approved and the uploaded file is deleted from R2. If it fails, the reason stays visible to the instructor for 7 days before it's cleaned up. A timer on the registry machine copies approved images to AWS ECR.
The rules I built it around
Nothing gets in another way.
Pushing needs a password and is only allowed from the node that runs scans. Workers can only pull, and the pull side is read-only. If one of those layers is misconfigured, the next one still holds.
Each step sees only its own secret.
The download link exists only in the fetch container, and the registry password only in publish. The backend never knows the registry password at all.
If it can't check, it doesn't pass.
Anything the scanner can't verify ends as an error, not a pass.
The scanner is hard to abuse.
It runs as non-root on a read-only filesystem, has no shell or package manager, calls tools without a shell, and its base images are pinned by digest.
A crash doesn't lose work.
All the state lives in the database, and one reconciler owns everything in Kubernetes. If the backend restarts halfway, the next round picks up where it stopped.
Choices and what they cost
A reconciler loop instead of starting jobs from the webhook
whyUploads still get accepted when the cluster is down, and only one piece of code ever touches Kubernetes.
costA scan can wait up to 15 seconds before it starts.
Fail only on fixable critical vulnerabilities
whyLab images are often built on large base images. A stricter rule would block almost everything.
costHigh-severity and unfixed critical issues pass with a warning. And a pass means no known problems, not proven safe.
Basic auth over plain HTTP to the registry
whyTraffic already runs inside Tailscale, which is encrypted.
costIt depends on the registry never being reachable outside the tailnet.
Things that went wrong
I lost the whole system once.
A git restore plus git clean wiped work I hadn't committed yet, and I had to rebuild it. Now I commit small pieces as soon as they work, and use git stash when I'm not sure.
A green test run that wasn't.
My first backend test run said 21 passed. It also said 115 skipped, because there was no database. Now I read the skipped number every time.
Check the real state before changing it.
I was about to rewrite our Tailscale rules because I thought they were outdated. The plan step showed they'd already been fixed. That's exactly why the script has a plan step.
Where it is now
- Status
- In development. The scan pipeline was tested end to end in October 2026; students don't use the platform yet.
- The gate
- Scanning is now the only way an instructor's image reaches the registry
- Next for my part
- Tests for the scan policy, stop retrying files that no longer exist, and keep a local copy of the Trivy database so scans don't depend on the internet
- Next for the platform
- More tools, and new services that give students more room to be creative