About the talk
This talk presents a novel way to integrate Gemini within game development workflows and runtime experiences on Unreal Engine.
The first facet focuses on enhancing productivity within the Unreal Editor. By leveraging multimodal capabilities, we propose a system that assists game developers with various tasks, from generating code snippets, and reasoning on visual programming and UI layouts to creating game assets and providing contextual design suggestions and debugging assistance. This integration aims to streamline the development process and significantly lower the barrier to entry for complex game creation.
The second aspect involves the automatic exposition of functions from Unreal Engine to the Gemini API through formalized function declarations. This allows Gemini agents to directly call and execute game functions at runtime, dynamically providing arguments as needed. This capability significantly reduces the cost of creating Agents for Agentic architectures powered by Gemini and unlocks unprecedented potential for game creation workflows as well as game runtime.
We will present real-world applications of these integrations, specifically detailing their use by the team behind Google Maps’ Immersive View.
Takeaway
– Attendees will take away how multimodal LLMs can help them on a variety of content creation tasks. Video game creation is a multimodal process and Multimodal LLMs, like Gemini, are now able to understand that context. Multimodal, unlocked their use in one of the most complex content creation field.
– Our integration was originally done within 48 hours and required no modification to the engine. Attendees will learn how to create an “agentic architecture” – powered by LLMs in any engine, with an example written for Unreal Engine.
– Attendees will get insights on the usage of LLMs within Google, specifically within the Immersive View team.
Experience level needed: Beginner, Intermediate, Advanced