Build your first LLM Agent 🤖

Let’s build an LLM agent that can generate visualisations using public data available on the Internet. You can find all the source code in this repository.

Why agents 🤖❓

Because Large Language Models alone are not enough to accurately answer complex tasks that require

• External information that was not present in the training dataset used to fit the LLM paramaters, or

• Many reasoning steps

For example, if you ask OpenAI gpt-3.5-turbo LLM to

Prompt: Create a plot with Python of the number of games won by the Golden State Warriors in each of the last 2 NBA seasons.

The response you get is

→ a perfectly valid Python code snippet, because GPT-3.5 was trained on vast amounts of Python code available on the Internet, and hence can work as a great code generator tool

import matplotlib.pyplot as plt

# Data for the number of games won by the Golden State Warriors in the last 2 NBA seasons
seasons = ['2019-2020', '2020-2021']
games_won = [15, 39]

# Create a bar plot
plt.figure(figsize=(10, 6))
plt.bar(seasons, games_won, color='blue')

# Add title and labels
plt.title('Number of Games Won by the Golden State Warriors in the Last 2 NBA Seasons')
plt.xlabel('Season')
plt.ylabel('Number of Games Won')

# Show plot
plt.show()

 

→ BUT, the data plotted is wrong, as it is not from the last 2 seasons, but from 2019 and 2020.

This happens because the NBA game data we need is not part of the training dataset used to fit GPT-3.5 model parameters (the model was last trained in November 2023).

So, to overcome this problem, you need to supercharge your LLM at least with

→ A tool to retrieve external data it needs to generate the answer (aka Retrieval Augmented Generation), and

→ The reasoning capabilities to digest the tool output, and decide whether the final answer is already there, or it needs to perform one more query to complete it.

Supercharging LLMs with these abilities is precisely what agents are all about.

LLM Agents 🤖

An agent is essentially a wrapper around your LLM, that provides extra functionality like

• Tool usage. The LLM is able to select and use tools, like internet search, to fetch relevant information it might need to accomplish the task.

• Multi-step reasoning. The LLM can generate a plan, execute it, and adjust it based on the partial outcomes obtained.

The LLM acts as a reasoning machine, that helps the agent choose the sequence of actions to take to solve the task.

Let me show you how to build a ReAct (Reason and Act) agent in Python that can generate the plot we want.

👉 You can find all the source code in this repository.

Step 1. Pick your LLM

You can either use

→ a local LLM running on your laptop with Ollama, or
→ connect to an external API like OpenAI or Cohere.

 

I experimented a bit with 3 models for this task:

1. Llama3 running locally on my laptop using Ollama,

2. OpenAI gpt-3.5-turbo, and

3. Cohere Command-R-plus

and the 3rd turned out to work best. Hence, if you wanna run the code on the repo you will need to first get an API key (for FREE) from Cohere.

Step 2. Define your tools 🛠️

In this case we need:

1. An Internet search tool.

   

    

2. A PythonREPL so the LLM can run Python code to debug its output

   

    

Step 3. Build the ReAct agent from the LLM you picked and the tools 

This step is super-simple thanks to a library like LangChain, which provides an implementation of the ReAct agent logic.

We first define the agent logic, using the

• tools we created

• the base LLM we picked, and

• an initial prompt of that can help us steer the the agent in the right direction

 

From this agent logic, we define the AgentExecutor, which is the runtime for an agent, that

• calls the agent

• executes the actions it chooses

• passes the action outputs back to the agent

• and repeats

We set verbose=True so we can see all intermediate steps the agent followed to reach the final output.

 

Step 4. Run the agent 🏃

We pass the input query

Prompt: Create a plot with Python of the number of games won by the Golden State Warriors in each of the last 2 NBA seasons.

to the agent_executor

 

and the agent start reasoning and acting.

• First, the agents plans its steps and prints them out on the console

  > Entering new AgentExecutor chain...
  
  I will search for the number of games won by the Golden State Warriors in the last two seasons. Then, I will use Python to create a plot of the data and save it as a PNG file.

• Then, takes its first action. In this case, runs an Internet search and realizes it cannot find the data it is looking for

  {'tool_name': 'internet_search', 'parameters': {'query': 'number of games won by golden state warriors in the last 2 seasons'}}
  
  I was unable to find the number of games won by the Golden State Warriors in the last two seasons. I will now search for the number of games won by the Golden State Warriors in the 2022-2023 and 2021-2022 seasons.

• Based on the previous step result, the agents runs 2 simpler web searches, on for each of the NBA seasons we want to plot. Both searches yield the desired results.

  {'tool_name': 'internet_search', 'parameters': {'query': 'number of games won by golden state warriors in 2022-2023 season'}}
  
  {'tool_name': 'internet_search', 'parameters': {'query': 'number of games won by golden state warriors in 2021-2022 season'}}
  
  I found that the Golden State Warriors won 44 games in the 2022-2023 season and 53 games in the 2021-2022 season. I will now use Python to create a plot of the data and save it as a PNG file.

• Given the data, the agent now uses the python_interpreter tool to generate the plot using matplotlib.

  {'tool_name': 'python_interpreter', 'parameters': {'code': 'import matplotlib.pyplot as plt\n\n# Data\ndata = {\n    "2022-2023": 44,\n    "2021-2022": 53\n}\n\n# Plot\nplt.bar(data.keys(), data.values())\nplt.xlabel("Season")\nplt.ylabel("Number of Games Won")\n\n# Save the plot\nplt.savefig("plots/warriors_victories.png")'}}
  
  Relevant Documents: 5,7,8,9,11,14,15
  Cited Documents: 15
  Answer: Here is a plot showing the number of games won by the Golden State Warriors in the last two seasons:
  
  ![Warriors Victories]("plots/warriors_victories.png")
  Grounded answer: Here is a plot showing the number of games won by the Golden State Warriors in the last two seasons:
  
  <co: 15>! [Warriors Victories]("plots/warriors_victories.png")</co: 15>

 

SEXY!

This is just one example. As you start experimenting with different input prompts, you will find scenarios in which the agent is not able to solve the task. Moving from agent prototypes like this, to production-ready agents is all about polishing rough edges, and ensuring the agent has access to the right tools.

But this is something we will leave for another day.

Wanna reproduce these results?

Simple, in 3 steps:

1. Clone this repository

2. make install

3. make run-agent