Web Scraping and Giphy API Integration#
In this tutorial, we will explore how to leverage the FastAgency framework to create a dynamic and interactive chatbot that integrates two powerful agents:
WebSurferAgent – A web-scraping agent capable of retrieving relevant content from webpages. GiphyAgent – An agent that interacts with the Giphy API to fetch GIFs based on the user’s request.
The chat system will operate between these two agents and the user, allowing them to scrape web content and generate GIFs based on that content, all within a seamless conversation. This tutorial will guide you through setting up these agents and handling user interaction in a secure, structured, and intuitive manner.
What You’ll Learn#
By the end of this tutorial, you’ll understand how to:
- Integrate external APIs like Giphy with FastAgency.
- Build and register agents that can autonomously scrape the web for relevant information.
- Use FastAgency workflows to manage agent interactions and user input.
- Present scraped content to the user and offer personalized GIF suggestions based on that content.
We will walk through setting up each agent, handling API security, and creating a cohesive conversation that can scrape data, process user input, and generate GIFs in response.
Let’s dive into creating a powerful interactive agent system with FastAgency!
Installation and API Key Setup#
Before we dive into building our agents, let’s go over the necessary setup. We will guide you through installing the FastAgency framework and obtaining the API key needed for the Giphy integration.
Installing FastAgency#
To get started, you need to install FastAgency with OpenAPI submodule. You can do this using pip, Python's package installer.
API Key Setup#
WebSurferAgent requires an Bing Web Search API key and GiphyAgent requires an API key to interact with Giphy's service. Follow these steps to create your API keys:
Create Bing Web Search API Key#
To create Bing Web Search API key, follow the guide provided.
Note
You will need to create Microsoft Azure Account.
Create a Giphy Account#
Step 1: If you don’t have a Giphy account, you’ll need to sign up:
- Go to Giphy Developers
- Click on Create Account.
Step 2: Navigate to Dashboard
Note
You may need to wait a few minutes after creating your account before being able to access the Dashboard page.
- Click on Create an API key and choose API as the type of app and give it a name (e.g., "FastAgency Giphy App").
- Agree to the terms and click Create App.
Step 3: Get Your API Key After creating the app, you’ll be provided with an API Key.
- Copy this key and continue with the following steps.
Set Up Your API Keys in the Environment#
To securely use the API keys in your project, you should store it in an environment variables.
You can set the API keys in your terminal as an environment variable:
Complete Application Code#
main.py
import os
from typing import Annotated, Any, Optional
from autogen import register_function
from autogen.agentchat import ConversableAgent
from fastagency import UI, FastAgency, Workflows
from fastagency.api.openapi.client import OpenAPI
from fastagency.api.openapi.security import APIKeyQuery
from fastagency.base import TextInput
from fastagency.runtime.autogen.agents.websurfer import WebSurferAgent
from fastagency.runtime.autogen.base import AutoGenWorkflows
from fastagency.ui.mesop import MesopUI
llm_config = {
"config_list": [
{
"model": "gpt-4o-mini",
"api_key": os.getenv("OPENAI_API_KEY"),
}
],
"temperature": 0.0,
}
openapi_url = "https://raw.githubusercontent.com/airtai/fastagency/refs/heads/main/examples/openapi/giphy_openapi.json"
giphy_api = OpenAPI.create(openapi_url=openapi_url)
giphy_api_key = os.getenv("GIPHY_API_KEY")
giphy_api.set_security_params(APIKeyQuery.Parameters(value=giphy_api_key))
GIPHY_SYSTEM_MESSAGE = """You are an agent in charge to communicate with the user and Giphy API.
Always use 'present_completed_task_or_ask_question' to interact with the user.
- make sure that the 'message' parameter contains all the necessary information for the user!
Initially, the Web_Surfer_Agent will provide you with some content from the web.
You must present this content provided Web_Surfer_Agent to the user by using 'present_completed_task_or_ask_question'.
Along with the content, ask the user if he wants you to generate some gifs based on the content.
- Do NOT generate gifs BEFORE you present the web content to the user, otherwise, you will be penalized!
Once get the wanted gifs, present them to the user by using 'present_completed_task_or_ask_question' again.
Note: Use '.gif' files when presenting a gif to the user and format it as a markdown gif -> 
- Also, make sure to add new lines '\n\n' between headlines and gifs for better readability.
e.g.:
'''
# Here are some gifs for you:
## Title 1
## Title 2
'''
Write 'TERMINATE' to end the conversation."""
wf = AutoGenWorkflows()
@wf.register(name="giphy_and_websurfer", description="Giphy and Websurfer chat")
def giphy_workflow_with_security(
wf: Workflows, ui: UI, initial_message: str, session_id: str
) -> str:
def is_termination_msg(msg: dict[str, Any]) -> bool:
return msg["content"] is not None and "TERMINATE" in msg["content"]
def present_completed_task_or_ask_question(
message: Annotated[str, "Message for examiner"],
) -> Optional[str]:
try:
msg = TextInput(
sender="giphy_agent",
recipient="giphy_agent",
prompt=message,
)
return ui.process_message(msg)
except Exception as e: # pragma: no cover
return f"present_completed_task_or_ask_question() FAILED! {e}"
giphy_agent = ConversableAgent(
name="Giphy_Agent",
system_message=GIPHY_SYSTEM_MESSAGE,
llm_config=llm_config,
human_input_mode="NEVER",
is_termination_msg=is_termination_msg,
)
web_surfer = WebSurferAgent(
name="Web_Surfer_Agent",
llm_config=llm_config,
summarizer_llm_config=llm_config,
human_input_mode="NEVER",
executor=giphy_agent,
is_termination_msg=is_termination_msg,
)
register_function(
present_completed_task_or_ask_question,
caller=giphy_agent,
executor=web_surfer,
name="present_completed_task_or_ask_question",
description="""Present completed task or ask question.
If you are presenting a completed task, last message should be a question: 'Do yo need anything else?'""",
)
functions = ["random_gif", "search_gifs", "trending_gifs"]
wf.register_api(
api=giphy_api,
callers=giphy_agent,
executors=web_surfer,
functions=functions,
)
chat_result = giphy_agent.initiate_chat(
web_surfer,
message=f"Users initial message: {initial_message}",
summary_method="reflection_with_llm",
max_turns=10,
)
return chat_result.summary # type: ignore[no-any-return]
app = FastAgency(wf=wf, ui=MesopUI(), title="Giphy and Websurfer chat")
Code Walkthrough#
Now we will go over each key part of the code, explaining its function and purpose within the FastAgency framework. Understanding these components is crucial for building a dynamic interaction between the user, the WebSurferAgent, and the GiphyAgent.
Creating the Giphy API Instance#
The following lines shows hot to initializes the Giphy API by loading the OpenAPI specification from a URL. The OpenAPI spec defines how to interact with the Giphy API, including endpoints, parameters, and security details.
Also, we configure the Giphy API with the GIPHY_API_KEY using set_security_params to authenticate our requests.
openapi_url = "https://raw.githubusercontent.com/airtai/fastagency/refs/heads/main/examples/openapi/giphy_openapi.json"
giphy_api = OpenAPI.create(openapi_url=openapi_url)
giphy_api_key = os.getenv("GIPHY_API_KEY")
giphy_api.set_security_params(APIKeyQuery.Parameters(value=giphy_api_key))
For more information, visit API Integration User Guide.
Registering the Workflow#
Here, we initialize a new workflow using AutoGenWorkflows() and register it under the name "giphy_and_websurfer". The @wf.register decorator registers the function to handle chat flow with security enabled, combining both GiphyAgent and WebSurferAgent.
@wf.register(name="giphy_and_websurfer", description="Giphy and Websurfer chat")
def giphy_workflow_with_security(
wf: Workflows, ui: UI, initial_message: str, session_id: str
) -> str:
Interaction with the user#
This is a core function used by the GiphyAgent to either present the task result or ask a follow-up question to the user. The message is wrapped in a TextInput object, and then ui.process_message() sends it for user interaction.
def present_completed_task_or_ask_question(
message: Annotated[str, "Message for examiner"],
) -> Optional[str]:
try:
msg = TextInput(
sender="giphy_agent",
recipient="giphy_agent",
prompt=message,
)
return ui.process_message(msg)
except Exception as e: # pragma: no cover
return f"present_completed_task_or_ask_question() FAILED! {e}"
Creating the Giphy and WebSurfer Agents#
- GiphyAgent: A ConversableAgent is created with the name "Giphy_Agent". It uses the system message defined earlier and relies on the termination function to end the chat when needed.
- WebSurferAgent: The WebSurferAgent is responsible for scraping web content and passes the retrieved data to the GiphyAgent. It’s configured with a summarizer to condense web content, which is useful when presenting concise data to the user. For more information, visit WebSurfer User Guide.
giphy_agent = ConversableAgent(
name="Giphy_Agent",
system_message=GIPHY_SYSTEM_MESSAGE,
llm_config=llm_config,
human_input_mode="NEVER",
is_termination_msg=is_termination_msg,
)
web_surfer = WebSurferAgent(
name="Web_Surfer_Agent",
llm_config=llm_config,
summarizer_llm_config=llm_config,
human_input_mode="NEVER",
executor=giphy_agent,
is_termination_msg=is_termination_msg,
)
Registering Functions#
The function present_completed_task_or_ask_question is registered to allow the GiphyAgent to ask questions or present completed tasks after receiving data from the WebSurferAgent.
register_function(
present_completed_task_or_ask_question,
caller=giphy_agent,
executor=web_surfer,
name="present_completed_task_or_ask_question",
description="""Present completed task or ask question.
If you are presenting a completed task, last message should be a question: 'Do yo need anything else?'""",
)
We specify which Giphy API functions can be used by the GiphyAgent: random_gif, search_gifs, and trending_gifs. These functions allow the agent to generate GIFs based on user input or trending content.
functions = ["random_gif", "search_gifs", "trending_gifs"]
wf.register_api(
api=giphy_api,
callers=giphy_agent,
executors=web_surfer,
functions=functions,
)
Initiating the Chat#
We initiate the conversation between the user, WebSurferAgent, and GiphyAgent. The user’s initial message is provided, and the system is configured to handle up to 10 turns of interaction. The conversation is summarized using the reflection_with_llm method, which uses a language model to summarize the chat.
Once the conversation ends, the summary is returned to the user, wrapping up the session.
chat_result = giphy_agent.initiate_chat(
web_surfer,
message=f"Users initial message: {initial_message}",
summary_method="reflection_with_llm",
max_turns=10,
)
return chat_result.summary # type: ignore[no-any-return]
Starting the Application#
The FastAgency app is created, using the registered workflows (wf) and web-based user interface (MesopUI). This makes the conversation between agents and the user interactive.
For more information, visit Mesop User Guide.
Running the Application#
Once the workflow is set up, you can run the application using the FastAgency CLI. Navigate to the directory where the script is located and run the following command:
╭─ Python package file structure ──╮
│ │
│ 📁 docs │
│ ├── 🐍 __init__.py │
│ └── 📁 docs_src │
│ ├── 🐍 __init__.py │
│ └── 📁 tutorial │
│ ├── 🐍 __init__.py │
│ └── 📁 giphy │
│ ├── 🐍 __init__.py │
│ └── 🐍 main.py │
│ │
╰──────────────────────────────────╯
/home/vscode/.local/lib/python3.10/site-packages/pydantic/_internal/_config.py:341: UserWarning: Valid config keys have changed in V2:
* 'keep_untouched' has been renamed to 'ignored_types'
warnings.warn(message, UserWarning)
[INFO] Importing autogen.base.py
/home/vscode/.local/lib/python3.10/site-packages/pydantic/main.py:214: UserWarning: A custom validator is returning a value other than `self`.
Returning anything other than `self` from a top level model validator isn't supported when validating via `__init__`.
See the `model_validator` docs (https://docs.pydantic.dev/latest/concepts/validators/#model-validators) for more details.
warnings.warn(
╭───────────── Importable FastAgency app ─────────────╮
│ │
│ from docs.docs_src.tutorial.giphy.main import app │
│ │
╰─────────────────────────────────────────────────────╯
[INFO] Creating MesopUI with import string: docs.docs_src.tutorial.giphy.main:app
[INFO] Starting MesopUI: import_string=docs.docs_src.tutorial.giphy.main:app, main_path=/tmp/tmpfi8uxdgv/main.py
Running with hot reload:
Running server on: http://localhost:32123
* Serving Flask app 'mesop.server.server'
* Debug mode: off
The command will launch a web interface where users can input their requests and interact with the agents (in this case http://localhost:32123)
Note
Ensure that your OpenAI API key is set in the environment, as the agents rely on it to interact using GPT-4o. If the API key is not correctly configured, the application may fail to retrieve LLM-powered responses.
Chat Example#
In this scenario, the user instructs the agents to scrape BBC Sport for the latest sports news.
Upon receiving the request, WebSurferAgent initiates the process by scraping the webpage for relevant updates.
Once the scraping is complete, the agents deliver their findings to the user. In the final step, the user asks for a few Premier League GIFs, which GiphyAgent promptly provides.
Conclusion#
In this tutorial, we walked through how to create a simple chatbot using FastAgency that can scrape web content and provide relevant GIFs. By integrating the WebSurferAgent and GiphyAgent, we built a tool that lets users gather information from the web and request GIFs all in one conversation.
You’ve learned how to:
- Set up and connect external APIs, like Giphy, to your project.
- Handle API key security.
- Build an interactive workflow where users can scrape content and get personalized responses.
- Offer dynamic, engaging conversations with added visuals like GIFs.
- With these skills, you now have the foundation to expand and add more features to your chatbot, making it even more interactive and useful.