New Computer Achieves 50% Higher Recall with LangSmith Integration
Felix Pinkston Jul 03, 2024 03:48
New Computer enhances memory retrieval, achieving 50% higher recall and 40% higher precision using LangSmith's advanced systems.
New Computer has significantly enhanced its memory retrieval system by integrating LangSmith, achieving a 50% higher recall rate and a 40% higher precision rate compared to its previous baseline, according to the LangChain Blog.
About New Computer
New Computer is the team behind Dot, the first personal AI designed to truly understand its users. Dot’s long-term memory system evolves by observing verbal and behavioral cues, creating a perception of true understanding by providing timely and personalized assistance.
A Brief Overview of Dot’s Agentic Memory
The innovative agentic memory system developed by New Computer dynamically creates or pre-calculates documents for future retrieval. Unlike standard retrieval-augmented generation (RAG) methods, this system structures information during memory creation, ensuring accurate and efficient retrieval as memories accumulate.
Dot’s memories include meta-fields such as status (e.g., COMPLETED or IN PROGRESS) and datetime fields like start or due dates, which serve as additional filters during high-frequency queries.
Improving Memory Retrieval with LangSmith
New Computer utilized LangSmith to iterate quickly on a dataset of labeled examples. To maintain user privacy, synthetic data was generated, creating synthetic users with LLM-generated backstories. The team stored queries and available memories in a LangSmith dataset, labeling relevant memories for each query and defining evaluation metrics like precision, recall, and F1.
Experiments began with a baseline system using semantic search to retrieve relevant memories. Various techniques were tested to assess performance, including similarity search and keyword methods like BM25. In some cases, pre-filtering by meta-fields was necessary for effective performance.
LangSmith’s SDK and Experiments UI allowed New Computer to run and evaluate these experiments efficiently, significantly improving their memory systems.
Adjusting the Conversation Prompt with LangSmith
Dot’s responses are generated by a dynamic conversational prompt, incorporating relevant memories, tool usage, and highly-contextual behavioral instructions. To optimize the prompt, synthetic users generated a wide range of queries, allowing the team to inspect the global effects of prompt changes using LangSmith’s experiment comparison view.
In failure cases, prompts were adjusted directly within the LangSmith UI, improving iteration speed while evaluating and adjusting conversation prompts.
What’s Next for New Computer
As New Computer aims to deepen human-AI relationships, they continue to enhance Dot’s ability to adapt to user preferences and provide bespoke experiences. With a recent launch bringing in a new wave of users, including a 45% conversion rate to the app’s paid tier, the partnership with LangChain and use of LangSmith remains pivotal in simulating complex human-AI interactions.
Image source: Shutterstock