Wednesday, October 23, 2024 – Banatao Auditorium in Sutardja Dai Hall, UC Berkeley Campus
Sky Camp is where you can get exposure to specific Sky Computing research projects through talks and demos, as well as hands-on experience with systems and technologies for emerging applications.
Agenda
9:00 AM – 10:00 AM: Breakfast
10:00 AM – 10:15 AM: Opening Talk: Overview of Sky Camp
10:15 AM – 12:30 PM: Session 1 – SkyPilot, DSPy, & Chatbot Arena
Chatbot Arena
An Open Platform for Evaluating LLMs by Human Preference
Large Language Models (LLMs) have unlocked new capabilities and applications; however, evaluating the alignment with human preferences still poses significant challenges. To address this issue, we introduce Chatbot Arena, an open platform for evaluating LLMs based on human preferences …
12:30 PM – 1:30 PM: Lunch
1:30 PM – 2:30 PM: Session 2 – R2E & LOTUS
R2E
Turning any GitHub Repository into a Programming Agent Environment
While Large Language Models’ coding capabilities have advanced rapidly, corresponding evaluation benchmarks on real-world programming setups are yet to catch up. Building a scalable and interactive testbed for evaluating general-purpose AI coding agents for real-world code has been challenging, particularly due to a lack of high-quality test suites available …
LOTUS
A Query Engine for Data Processing with LLMs
The powerful semantic capabilities of modern language models (LMs) create exciting opportunities for building AI-based analytics systems that reason over vast knowledge corpora. A wide variety of applications require a form of bulk semantic processing, where the analytics system must process large amounts of data and apply semantic-based analysis across the whole dataset …
2:30 PM – 2:45 PM: Break
2:45 PM – 4:00 PM: Session 3 – Compass & Gorilla
Compass
Encrypted Semantic Search with High Accuracy
We introduce Compass, a semantic search system over encrypted data that offers high accuracy, comparable to state-of-the-art plaintext search algorithms while protecting data, queries and search results from a fully compromised server. Additionally, Compass enables privacy-preserving RAG where both the RAG database and the query are protected.
Gorilla
Large Language Model Connected with Massive APIs
Large Language Models (LLMs) have seen an impressive wave of advances recently, with models now excelling in a variety of tasks, such as mathematical reasoning and program synthesis. However, their potential to effectively use tools via API calls remains unfulfilled. This is a challenging task even for today’s state-of-the-art LLMs such as GPT-4, largely due to their inability to generate accurate input arguments and their tendency to hallucinate the wrong usage of an API call …
4:00 PM – 4:15 PM: Break
4:15 PM – 5:30 PM: Session 4 – vLLM & MemGPT
vLLM
A High-Throughput and Memory-Efficient Inference and Serving Engine for LLMs
High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size …
MemGPT
Building Persistent LLM Agents with Long-Term Memory
Large language models (LLMs) have revolutionized AI, but are constrained by limited context windows, hindering their utility in tasks like extended conversations and document analysis. To enable using context beyond limited context windows, we propose virtual context management, a technique drawing inspiration from hierarchical memory systems in traditional operating systems which provide the illusion of an extended virtual memory via paging between physical memory and disk …
FAQ
Is there parking?
We strongly encourage public transportation or ride-sharing. However, we can provide parking passes for the nearby garage, Upper Hearst Parking Structure. The garages tend to fill up by around 10:00 AM, so please arrive early. If you will need a parking pass, please email us.
How do I register?
Registration is by invitation only. Invites will be sent beginning in mid-September.
Prerequisites / Technical Requirements
The ideal participant would be a practitioner and/or engineer who would like to gain experience using our software, preferably with some project management background.
- Laptop
- “Modern” browser (Firefox, Chrome, Safari etc.)
- Experience programming in Python in notebook environment
- Basic understanding of AI/ML concepts (e.g., training, validation, linear models)
Lodging
Some local hotel options are listed on our contact page.
Contact
For questions, email skycamp [at] berkeley.edu