Digital Twin

Overview

The MakerSpace Digital Twin is a real-time monitoring and management platform built as a Gannon University Senior Design project for Lockheed Martin. It tracks the status of Bambu Lab 3D printers (live and simulated), monitors personnel occupancy using computer vision, manages filament inventory, and exposes everything through a secure web portal that embeds a live 3D model of the physical makerspace. An AI agent powered by a locally-hosted LLM lets operators ask natural language questions about printer state, scheduled jobs, and inventory — with the AI avatar responding directly inside the 3D twin.

The system is fully modular: each capability is its own service and repository so teams can run the core backend alone or incrementally layer in CV and AI features.

Team: Matthew Gentry, Phu Ly, Khang Mai, Emilio Bell, Nathan Collins

---

Repositories

Backend API

GitHub: basic-makerspace-digital-twin-backend

The core data hub for the entire system. Built with C# / .NET 8 and ASP.NET Core, it handles all printer state management, job scheduling, telemetry streaming, and CV event consumption.

Stack: .NET 8, ASP.NET Core, C#, PostgreSQL, InfluxDB, RabbitMQ, Redis, Entity Framework Core

Key responsibilities:

• REST API for printer registry, task history, scheduled print jobs, and inventory state
• SSE (Server-Sent Events) streaming of real-time telemetry per printer — chosen over WebSockets because it is simpler, firewall-friendly, and sufficient for one-way server-push
• FleetCache — in-memory composite store per device that absorbs read pressure so Unity and the frontend don't hammer the database on the hot path
• CvEventsConsumerWorker consumes RabbitMQ events from the CV inventory pipeline and updates PostgreSQL inventory state
• PrinterSimulationTelemetryWorker generates simulated telemetry and publishes it via InMemoryTelemetryPublisher for development and demo use
• Bambu Proxy captures device-initiated telemetry streams from physical Bambu Lab printers, normalizes them into a validated Contract DTO, and forwards them into the StateSync pipeline
• InfluxDB stores high-frequency time-series telemetry (nozzle temp, bed temp, layer progress) — PostgreSQL stores relational/historical data

---

3D Digital Twin (Unity WebGL)

GitHub: Digital-Twin-WebGL

The interactive 3D frontend rendered in a web browser via Unity 6 and WebGL. Operators can navigate the virtual makerspace, select printers, view live telemetry dashboards, and interact with an AI NPC avatar.

Stack: Unity 6 (6000.4.0f1), C#, Universal Render Pipeline (URP), WebGL, Convai SDK, custom JSLib SSE Bridge, Brotli compression

Key responsibilities:

• Real-time 3D visualization of the makerspace floor with printer operational state (Running/Idle) updated every 60 seconds via REST fleet sync
• Per-printer telemetry sidebar showing nozzle temperature, bed temperature, and print progress — streamed live via SSE
• Custom SSEBridge.jslib JavaScript plugin bridges Unity C# calls to the browser's native EventSource API — required because standard C# networking (HttpClient, SseClient) does not work in WebGL browser contexts due to threading and CORS restrictions
• Convai SDK integration provides a conversational AI NPC capable of answering printer status questions and executing spatial commands: Highlight, Display Printer Dashboard, Show Me, Move To
• FilamentManager moves 3D spool objects between Storage and Active zones in response to CV inventory zone update events from the backend
• WASD navigation, drag-and-drop machine repositioning with collision detection, coordinate persistence to PostgreSQL — with local hold-and-sync on connection loss
• Custom ServiceWorker.js injects Cross-Origin-Embedder-Policy and Cross-Origin-Opener-Policy headers required for SharedArrayBuffer (which Unity WebGL depends on)

---

Web Portal (Frontend)

GitHub: WebsiteWrapperFrontend

The secure web portal that is the entry point and security boundary for the entire system. Users log in here and access all features — the Unity embed, personnel canvas, inventory calculators, and AI chat.

Stack: Node.js (≥20.0.0), Express 4.19.2, JWT (jsonwebtoken 9.0.2), Helmet.js, HTML5 Canvas, Vercel

Key responsibilities:

• JWT authentication with HttpOnly cookies — session tokens are stored in HttpOnly cookies, preventing JavaScript from reading them (XSS defense)
• Two-factor authentication (2FA) flow: login → backend challenge → 6-digit code entry → JWT issued
• Role-based access control via requireAuth and requireRole() middleware
• Unity WebGL files gated behind authentication — src/middleware/unity.js validates JWT before serving any .unityweb or WASM files
• Helmet.js applies 14+ security headers automatically including CSP, X-Frame-Options, HSTS, and custom COOP/COEP/CORP headers required for Unity WebGL's SharedArrayBuffer
• Personnel canvas: polls CV tracking API every 6 seconds, samples 3 readings per cycle, uses the majority-vote count per zone to filter noise, renders occupancy on an HTML5 Canvas floor plan overlay
• Proxies MJPEG video streams from the CV personnel service to resolve CORS and mixed-content restrictions in the browser
• Inventory management calculators: EOQ (Economic Order Quantity), ROP (Reorder Point), TIC (Total Inventory Cost)
• AI chat panel that forwards operator queries to the agent service

---

CV Personnel Tracking

GitHub: CV-Personnel-Tracking

Detects and counts people in each defined zone of the MakerSpace using a live camera feed and YOLOv8 object detection, publishing zone occupancy counts to the web portal.

Stack: Python 3.8+, FastAPI, YOLOv8 (yolov8n.pt), OpenCV, Cloudflare Tunnel

Key responsibilities:

• Real-time person detection from a camera feed: YOLOv8 inference → bounding boxes + class labels → zone membership check (config/zones.json) → count per zone → POST to backend API
• Publishes zone occupancy counts within 5 seconds of a person entering or leaving a zone
• Streams a live MJPEG video feed from the tracking camera through the portal
• Cloudflare Tunnel exposes the local tracking API publicly without requiring a static IP or port forwarding — the tunnel provides a consistent public URL regardless of local network configuration
• Model trained with custom dataset labeled via Roboflow Annotate, split with scikit-learn; yolov8n (nano) chosen for real-time inference speed on limited hardware

---

CV Inventory Tracking

GitHub: CV-Inventory-Tracking

Detects filament spool locations within workstation zones using computer vision and manages inventory state, with a human-in-the-loop confirmation model for uncertain detections.

Stack: Python 3.8+, FastAPI, YOLOv8 (yolov8n.pt / yolov8s.pt), OpenCV, Pillow, RabbitMQ, QR codes, Roboflow Annotate, scikit-learn

Key responsibilities:

• Spool detection pipeline: camera frame → YOLOv8 inference → zone membership assignment → event classification (APPEAR / DISAPPEAR / TRANSFER) → publish zone update event to backend via RabbitMQ
• Human-in-the-loop confirmation: CV detections above confidence threshold are auto-committed; uncertain detections require an operator to scan the spool's QR code to confirm before the inventory record is updated — prevents false-positive zone transfers from corrupting inventory state
• QR code generation per filament spool using Pillow and the qrcode package
• REST API for querying and managing zones, spools, and printers in the inventory system
• Publishes events to RabbitMQ so the .NET backend remains the authoritative inventory state owner; the Unity twin reflects changes by repositioning 3D spool objects
• Dataset labeled with Roboflow Annotate, split with scikit-learn; supports retraining via scripts in /training

---

AI Agent Service

GitHub: makerspace-digital-twin-agent-service

A local LLM agent service that answers natural-language questions about the printing farm. All inference runs on-premises via Ollama — no printer telemetry or operational data is sent to external cloud services.

Stack: Python, FastAPI, Ollama (qwen2.5:7b-instruct), sentence-transformers (all-MiniLM-L6-v2), LangGraph-style orchestration, FAISS (RAG index), Docker

Key responsibilities:

• Accepts natural-language queries from the web portal AI chat panel or the Convai Unity avatar
• Queries the .NET backend REST API for live printer state, active tasks, and scheduled job queue to ground responses in current system data
• Retrieval-Augmented Generation (RAG) over a static knowledge index pre-built at startup — provides factual context about MakerSpace operations and equipment without requiring the LLM to memorize domain facts, reducing hallucination
• qwen2.5:7b-instruct model chosen for strong instruction-following and structured reasoning at 7B parameter scale; all-MiniLM-L6-v2 embeddings run on CPU for fast real-time retrieval
• Local LLM (Ollama) chosen over cloud APIs (OpenAI, Anthropic) for privacy (operational data stays on-premises), zero API cost, no rate limit dependency, and offline operability
• Returns responses formatted for the Convai conversational AI interface, enabling the Unity AI avatar to speak live system state
• Runs on port 9000; containerized with Docker Compose alongside Ollama

---

Architecture

Frontend to Backend: The Express web portal proxies all authentication and data requests to the .NET backend. Users log in through the portal, which validates credentials and stores a JWT cookie. All dashboard data (printer list, task history, scheduled jobs) is fetched from the backend REST API. The browser never communicates directly with the backends — the portal is the security boundary.

Unity to Backend: The Unity WebGL twin connects to the .NET backend via two channels — a REST call every 60 seconds to sync the full printer fleet state, and a persistent SSE connection per printer for real-time telemetry. A custom SSEBridge.jslib JavaScript plugin is required because standard C# networking does not function in WebGL browser contexts.

CV Inventory to Backend: When the CV pipeline detects a filament spool moving between zones, it publishes an event to RabbitMQ. The backend's CvEventsConsumerWorker picks up the event, updates PostgreSQL inventory state, and the Unity twin reflects the change by moving 3D spool objects.

CV Personnel to Frontend: The web portal polls the CV-Personnel-Tracking API every 6 seconds (sampling 3 times and using the majority-vote count per zone) and renders occupancy on an HTML5 Canvas overlay of the makerspace floor plan.

Frontend to Agent Service: When an operator asks a question in the AI chat panel, the frontend sends it to the agent service. The agent queries the backend for live data, retrieves context from its static knowledge base via RAG, and returns an answer formatted for the Convai Unity avatar.

Key Challenges

Real-time state at scale: Machine shadows change many times per second. The FleetCache (in-memory composite store per device) absorbs read pressure so neither the Unity frontend nor the orchestrator layer hammers the database on the hot read path. SSE was chosen over WebSockets because it is simpler, firewall-friendly, and sufficient for server-push scenarios where the client never needs to send data back on the same channel.

WebGL networking constraints: Standard C# networking APIs (HttpClient, SseClient) do not work in a WebGL browser context. A custom JavaScript plugin (SSEBridge.jslib) was written to bridge Unity C# calls to the browser's native EventSource API, enabling real-time SSE from within the 3D environment.

Bambu printer telemetry: Bambu printers push telemetry from the device rather than responding to polling. A custom Bambu Proxy service captures this device-initiated stream, normalizes it into a validated Contract DTO, and forwards it to the backend's StateSync pipeline — isolating the physical device's quirks from the rest of the system.

Local LLM agent with RAG: The AI agent service uses LangGraph to orchestrate a qwen2.5:7b model running locally via Ollama. A static knowledge index (pre-built at startup) provides RAG context so the agent can answer questions about the printing farm without hallucinating. The agent queries the live backend API for current printer state and combines that with the RAG context to produce grounded answers.

Two CV pipelines, shared port, different concerns: Personnel tracking and inventory tracking are separate FastAPI services with distinct YOLOv8 models, zone configurations, and output sinks. Personnel counts are pushed directly to the frontend; inventory events are published to RabbitMQ so the backend remains the authoritative state owner.

Drag-and-drop with persistence: When a user repositions a machine in the Unity scene, the system validates the new placement (collision detection, space constraints) and persists the coordinates to PostgreSQL. If the connection is lost mid-drag, coordinates are held locally and synchronized on reconnect so no layout changes are silently discarded.

Human-in-the-loop inventory confirmation: CV models produce false positives. The inventory pipeline treats detections as signals, not facts — uncertain zone transfer events require an operator QR scan to confirm before the inventory record is updated. High-confidence detections above threshold are auto-committed. This design preserves inventory data integrity without requiring perfect CV accuracy.

Outcomes

All core features are functional: real-time printer status visualization in the 3D Unity twin, layout drag-and-drop with coordinate persistence, historical telemetry dashboards per machine, CV-based personnel zone occupancy, CV-based filament spool inventory management via RabbitMQ, a web portal with authentication and AI chat, and an AI operator assistant with RAG-backed answers. The system meets the project targets of 30 FPS minimum in the Unity scene and sub-2-second latency on real-time data updates.