How to Build GPT-4 Vision AI Agents with AutoGen

Since the OpenAI Dev Day, you can use GPT-4V over OpenAI API. It means, AutoGen now supports Multimodal AI Agents powered by GPT-4 Vision. In this AutoGen tutorial, you’ll learn how to create vision AI Agents with OpenAI API and GPT-4 Vision.


🤝 Connect with me 🤝
LinkedIn: https://www.linkedin.com/in/kris-ograbek/
Medium: https://medium.com/@kris-ograbek


+++ Useful Resources +++
Notebook for this project: https://github.com/krisograbek/autogen-tutorials/blob/main/AutoGen-GPT-4V.ipynb

+++ OpenAI API +++
Vision Models: https://platform.openai.com/docs/guides/vision

+++ AutoGen Vision Agents +++
Blog post from AutoGen: https://microsoft.github.io/autogen/blog/2023/11/06/LMM-Agent/
Example notebook: https://github.com/microsoft/autogen/blob/main/notebook/agentchat_lmm_gpt-4v.ipynb


+++ AutoGen Sources +++
Docs: https://microsoft.github.io/autogen/docs/Examples/AutoGen-AgentChat
GitHub: https://github.com/microsoft/autogen/tree/main
Official Paper: https://arxiv.org/abs/2308.08155
Multi-agent Conversation Framework: https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat/
SDK: https://microsoft.github.io/autogen/docs/reference/agentchat/conversable_agent/


Chapters:
0:00 Intro
1:32 Import config
2:14 GPT-4 Vision AI Agent and a new Agent type
3:45 Asking GPT-4V to describe an image
4:32 GPT-4V Agent compares images
6:08 GPT-4V Agent reads facial expression
6:53 Outro