MIT Labnetwork Nanobot (GenAI Knowledge System)
GenAI Applications • Research Infrastructure • Scientific Communities
Live Application
Try the Nanobot web application: nanobot.chat.
Overview
A retrieval-augmented GenAI system designed to recover and operationalize decades of institutional knowledge embedded in the MIT Labnetwork technical forum. The Nanobot enables natural-language access to historical discussions, patterns, and expertise across the global nanofabrication community without requiring new content creation or changes to existing workflows.
Challenge
For nearly 30 years, the MIT Labnetwork mailing list has functioned as the primary technical forum for the global micro- and nanofabrication facility community. Facility directors, engineers, and administrators from a small and highly specialized field have used it to exchange operational knowledge that rarely appears in publications: tool failure modes, process edge cases, vendor behavior, safety incidents, and lessons learned from running complex shared facilities.
Over time, this accumulated into thousands of messages distributed across email threads and monthly web archives. Although the knowledge was extremely high value, it was effectively inaccessible. Historical discussions were difficult to search, institutional memory was lost as staff turned over, and the same questions were repeatedly asked years apart. The community had unknowingly built a rich knowledge base—but one locked inside a legacy communication format.
The core problem was not generating new knowledge, but recovering and making usable what already existed.
System Design
The MIT LabNetwork Nanobot is a Retrieval-Augmented Generation (RAG) system that enables natural-language exploration of historical expert discussions from the forum archive. The system is designed to augment, not replace, expert judgment by allowing users to retrieve relevant past conversations, patterns, and contributors in response to technical and operational questions.
A central design principle was compatibility with existing data and workflows. The Nanobot operates entirely on archival content, requiring no new content creation from the community and no changes to how discussions were originally conducted. Generative AI functions strictly as an interface layer over historical data, improving access to institutional knowledge without altering its provenance.
System Architecture Highlights
Data Aggregation and Cleaning
The LabNetwork forum maintains a publicly accessible web archive indexed by month dating back to 2007, with older content preserved in email form. A custom scraping pipeline was developed to ingest the full archive. Raw messages contained significant noise, including quoted replies, signatures, headers, and formatting artifacts that would degrade retrieval quality.
Messages were programmatically cleaned to retain only essential information: sender identity, timestamps, thread relationships, subject lines, and message bodies. A large language model was then used as a second-stage cleaner to further normalize message text and remove residual non-semantic content.
The resulting dataset preserved both the semantic content of the discussions and the relational structure of the forum.
Knowledge Representation and Storage
Cleaned message bodies and subject lines were vectorized using an OpenAI embedding model and stored in a PostgreSQL database with pgvector support. Metadata—including sender, institution (where available), message IDs, and thread IDs—was stored alongside the embeddings.
This hybrid representation enabled both semantic retrieval and traditional relational queries over the same corpus, which proved critical for supporting different classes of user questions.
Retrieval Modes and Query Strategies
The system supports multiple retrieval strategies to accommodate different types of user questions and analytical needs.
Vector Retrieval: Supports open-ended technical questions by embedding user prompts and identifying historically relevant discussions via semantic similarity. Retrieved messages are passed to a language model to generate responses grounded in prior expert conversations rather than speculative output.
SQL Database Retrieval: Enables analytical and meta-questions about the dataset itself—such as participation patterns, institutional activity, or temporal trends—by translating natural-language queries into SQL executed directly against the relational database.
Thread Retrieval: Preserves full conversational context by retrieving complete discussion threads in chronological order once a relevant topic or message is identified.
Together, these modes allow users to explore the archive by meaning, structure, or historical sequence, depending on their needs.
Outcomes
The Nanobot was released publicly at the 2024 UGIM International Conference at MIT, where it was demonstrated live to nanofabrication leaders from around the world. The response was strongly positive, particularly from facility staff who recognized long-forgotten discussions and solutions resurfacing through the system.
This project provided the first concrete demonstration to the Labnetwork community that generative AI could be used to recover institutional knowledge rather than replace expertise. It reframed AI as a tool for preserving operational memory, reducing redundant problem-solving, and accelerating decision-making across shared facilities.
More broadly, the system validated a scalable approach for applying RAG techniques to legacy, unstructured research infrastructure data—data that is highly contextual, difficult to formalize, and extremely valuable. The system has since been used by facilities worldwide as a reference point for applying generative AI to shared research infrastructure.
Future Plans
The Labnetwork archive is refreshed on a monthly cadence to incorporate new discussions. Future development will expand nanobot.chat into a multi-application platform, hosting additional GenAI tools that demonstrate practical uses of AI for the nanofabrication community, including documentation search, operational analytics, and cross-facility knowledge sharing.