Skip to content

Quick start#

You can try out FastAgency to build and deploy your multi-agent application in less than one hour by following this guide. You will learn how to write you workflow, test and debug it and finally deploy it on cloud. The deployed application will have chat interface.

You can use three different setups, depending on how scalable solution you need:

  • Mesop: This setup uses Mesop to build a web application for interacting with our workflow. It supports a single-worker deployments only, limiting its scalability. However, it is the fastest way to build and test your application.

  • FastAPI + Mesop: This is fairly scalable setup using FastAPI to execute your workflows and Mesop for interactive web application. FastAPI supports execution with multiple workers, with each workflow being executed in the context of a WebSocket connection. Mesop is still limited to a single worker, although there is much less load on it due to workflows being executed in the FastAPI workers.

  • NATS + FastAPI + Mesop: This is the most scalable setup using a distributed message broker NATS.io MQ. Workflows are being executed with multiple workers that attach to the MQ waiting for initiate workflow messages. Such workers can be running on different machines or even different data centers/cloud providers. Message queues are highly scalable, but more difficult to integrate with end-clients. In order to make such integrations easier, we will connect our NATS-based message queue with the FastAPI application.

Project setup#

We strongly recommend using Cookiecutter for setting up a FastAgency project. It creates the project folder structure, default workflow, automatically installs all the necessary requirements, and creates a devcontainer that can be used with Visual Studio Code for development.

You could also use virtual environment managers such as venv, and a Python package manager, such as pip.

  1. Install Cookiecutter with the following command:

    pip install cookiecutter
    

  2. Run the cookiecutter command:

    cookiecutter https://github.com/airtai/cookiecutter-fastagency.git
    

  3. Depending on the type of the project, choose the appropriate option in step 3:

    [1/5] project_name (My FastAgency App):
    [2/5] project_slug (my_fastagency_app):
    [3/5] Select app_type
        1 - fastapi+mesop
        2 - mesop
        3 - nats+fastapi+mesop
        Choose from [1/2/3] (1): 2
    [4/5] Select python_version
        1 - 3.12
        2 - 3.11
        3 - 3.10
        Choose from [1/2/3] (1):
    [5/5] Select authentication
        1 - basic
        2 - google
        3 - none
        Choose from [1/2/3] (1):
    

    This command installs FastAgency with support for both the Console and Mesop interfaces for AutoGen workflows.

    [1/5] project_name (My FastAgency App):
    [2/5] project_slug (my_fastagency_app):
    [3/5] Select app_type
        1 - fastapi+mesop
        2 - mesop
        3 - nats+fastapi+mesop
        Choose from [1/2/3] (1): 1
    [4/5] Select python_version
        1 - 3.12
        2 - 3.11
        3 - 3.10
        Choose from [1/2/3] (1):
    [5/5] Select authentication
        1 - basic
        2 - google
        3 - none
        Choose from [1/2/3] (1):
    

    This command installs FastAgency with support for both the Console and Mesop interfaces for AutoGen workflows, with FastAPI handling input requests and workflow execution.

    [1/5] project_name (My FastAgency App):
    [2/5] project_slug (my_fastagency_app):
    [3/5] Select app_type
        1 - fastapi+mesop
        2 - mesop
        3 - nats+fastapi+mesop
        Choose from [1/2/3] (1): 3
    [4/5] Select python_version
        1 - 3.12
        2 - 3.11
        3 - 3.10
        Choose from [1/2/3] (1):
    [5/5] Select authentication
        1 - basic
        2 - google
        3 - none
        Choose from [1/2/3] (1):
    

    This command installs FastAgency with support for both the Console and Mesop interfaces for AutoGen workflows, with FastAPI serving input and independent workers communicating over the NATS.io protocol workflows. This is the most scable setup, recommended for large production workloads.

  4. Executing the cookiecutter command will create the following file structure:

    my_fastagency_app
    ├── docker
    │   ├── content
    │   │   ├── nginx.conf.template
    │   │   └── run_fastagency.sh
    │   └── Dockerfile
    ├── my_fastagency_app
    │   ├── deployment
    │   │   ├── __init__.py
    │   │   └── main.py
    │   ├── local
    │   │   ├── __init__.py
    │   │   ├── main_console.py
    │   │   └── main_mesop.py
    │   ├── __init__.py
    │   └── workflow.py
    ├── scripts
    │   ├── build_docker.sh
    │   ├── check-registered-app-pre-commit.sh
    │   ├── check-registered-app.sh
    │   ├── deploy_to_fly_io.sh
    │   ├── lint-pre-commit.sh
    │   ├── lint.sh
    │   ├── register_to_fly_io.sh
    │   ├── run_docker.sh
    │   ├── run_mesop_locally.sh
    │   ├── static-analysis.sh
    │   └── static-pre-commit.sh
    ├── tests
    │   ├── __init__.py
    │   ├── conftest.py
    │   └── test_workflow.py
    ├── README.md
    ├── fly.toml
    └── pyproject.toml
    
    my_fastagency_app
    ├── docker
    │   ├── content
    │   │   ├── nginx.conf.template
    │   │   └── run_fastagency.sh
    │   └── Dockerfile
    ├── my_fastagency_app
    │   ├── deployment
    │   │   ├── __init__.py
    │   │   ├── main_1_fastapi.py
    │   │   └── main_2_mesop.py
    │   ├── local
    │   │   ├── __init__.py
    │   │   ├── main_console.py
    │   │   └── main_mesop.py
    │   ├── __init__.py
    │   └── workflow.py
    ├── scripts
    │   ├── build_docker.sh
    │   ├── check-registered-app-pre-commit.sh
    │   ├── check-registered-app.sh
    │   ├── deploy_to_fly_io.sh
    │   ├── lint-pre-commit.sh
    │   ├── lint.sh
    │   ├── register_to_fly_io.sh
    │   ├── run_docker.sh
    │   ├── run_mesop_locally.sh
    │   ├── static-analysis.sh
    │   └── static-pre-commit.sh
    ├── tests
    │   ├── __init__.py
    │   ├── conftest.py
    │   └── test_workflow.py
    ├── README.md
    ├── fly.toml
    └── pyproject.toml
    
    my_fastagency_app
    ├── docker
    │   ├── content
    │   │   ├── nginx.conf.template
    │   │   └── run_fastagency.sh
    │   └── Dockerfile
    ├── my_fastagency_app
    │   ├── deployment
    │   │   ├── __init__.py
    │   │   ├── main_1_nats.py
    │   │   ├── main_2_fastapi.py
    │   │   └── main_3_mesop.py
    │   ├── local
    │   │   ├── __init__.py
    │   │   ├── main_console.py
    │   │   └── main_mesop.py
    │   ├── __init__.py
    │   └── workflow.py
    ├── scripts
    │   ├── build_docker.sh
    │   ├── check-registered-app-pre-commit.sh
    │   ├── check-registered-app.sh
    │   ├── deploy_to_fly_io.sh
    │   ├── lint-pre-commit.sh
    │   ├── lint.sh
    │   ├── register_to_fly_io.sh
    │   ├── run_docker.sh
    │   ├── run_mesop_locally.sh
    │   ├── static-analysis.sh
    │   └── static-pre-commit.sh
    ├── tests
    │   ├── __init__.py
    │   ├── conftest.py
    │   └── test_workflow.py
    ├── README.md
    ├── fly.toml
    └── pyproject.toml
    
  5. To run LLM-based applications, you need an API key for the LLM used. The most commonly used LLM is OpenAI. To use it, create an OpenAI API Key and set it as an environment variable in the terminal using the following command:

    export OPENAI_API_KEY=openai_api_key_here
    

    If you want to use a different LLM provider, follow this guide.

    Alternatively, you can skip this step and set the LLM API key as an environment variable later in the devcontainer's terminal. If you open the project in Visual Studio Code using GUI, you will need to manually set the environment variable in the devcontainer's terminal.

  6. Open the generated project in Visual Studio Code with the following command:

    code my_fastagency_app
    

  7. Once the project is opened, you will get the following option to reopen it in a devcontainer:

  8. After reopening the project in devcontainer, you can verify that the setup is correct by running the provided tests with the following command:

    pytest -s
    

    You should get the following output if everything is correctly setup.

    =================================== test session starts ===================================
    platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0
    rootdir: /workspaces/my_fastagency_app
    configfile: pyproject.toml
    plugins: asyncio-0.24.0, anyio-4.6.2.post1
    asyncio: mode=Mode.STRICT, default_loop_scope=None
    collected 1 item
    
    tests/test_workflow.py .                                                            [100%]
    
    ==================================== 1 passed in 1.02s ====================================
    

    Running the test could take up to 30 seconds, depending on latency and throughput of OpenAI (or other LLM providers).

Info

If you used a different project_slug than the default my_fastagency_app this will be reflected in the project module naming. Keep this in mind when running the commands further in this guide (in Run Application), you will need to replace my_fastagency_app with your project_slug name.


Workflow Development#

Define the Workflow#

You need to define the workflow that your application will use. This is where you specify how the agents interact and what they do. Here's a simple example of a workflow definition as it is generated by the cookie cutter under my_fastagency_app/workflow.py:

import os
from typing import Any

from autogen.agentchat import ConversableAgent
from fastagency import UI
from fastagency.runtimes.autogen import AutoGenWorkflows

llm_config = {
    "config_list": [
        {
            "model": "gpt-4o-mini",
            "api_key": os.getenv("OPENAI_API_KEY"),
        }
    ],
    "temperature": 0.8,
}

wf = AutoGenWorkflows()


@wf.register(name="simple_learning", description="Student and teacher learning chat")  # type: ignore[misc]
def simple_workflow(ui: UI, params: dict[str, Any]) -> str:
    initial_message = ui.text_input(
        sender="Workflow",
        recipient="User",
        prompt="I can help you learn about mathematics. What subject you would like to explore?",
    )

    student_agent = ConversableAgent(
        name="Student_Agent",
        system_message="You are a student willing to learn.",
        llm_config=llm_config,
    )
    teacher_agent = ConversableAgent(
        name="Teacher_Agent",
        system_message="You are a math teacher.",
        llm_config=llm_config,
    )

    chat_result = student_agent.initiate_chat(
        teacher_agent,
        message=initial_message,
        summary_method="reflection_with_llm",
        max_turns=3,
    )

    return str(chat_result.summary)

This code snippet sets up a simple learning chat between a student and a teacher. It defines the agents and how they should interact and specify how the conversation should be summarized.

Run and Debug the Workflow#

To ensure that the workflow we have defined is working properly, we can test it locally using MesopUI. The code below can be found under my_fastagency_app/local/main_mesop.py and imports the defined workflow and sets up MesopUI:

from fastagency import FastAgency
from fastagency.ui.mesop import MesopUI

from ..workflow import wf

app = FastAgency(
    provider=wf,
    ui=MesopUI(),
    title="My FastAgency App",
)

# start the fastagency app with the following command
# gunicorn my_fastagency_app.local.main_mesop:app

Run MesopUI locally with the following command:

Terminal

gunicorn my_fastagency_app.local.main_mesop:app

Terminal

waitress-serve --listen=0.0.0.0:8000 my_fastagency_app.local.main_mesop:app

Open the MesopUI URL http://localhost:8000 in your browser. You can now use the graphical user interface to start, run, test and debug the autogen workflow manually.

Initial message

Run Workflow Tests#

We can also use pytest to test the autogen workflow automatically, instead of manually testing it using MesopUI by using the generated pytest test found under tests/test_workflow.py.

from uuid import uuid4

import pytest
from fastagency.ui.console import ConsoleUI

from my_fastagency_app.workflow import wf
from tests.conftest import InputMock


def test_workflow(monkeypatch: pytest.MonkeyPatch) -> None:
    monkeypatch.setattr("builtins.input", InputMock([""] * 5))

    result = wf.run(
        name="simple_learning",
        ui=ConsoleUI().create_workflow_ui(workflow_uuid=uuid4().hex),
    )

    assert result is not None

Run the test with the following command:

pytest -s

Running the test could take up to 30 seconds, depending on latency and throughput of OpenAI (or other LLM providers).

Deployment files#

Depending on the interface you choose when setting up the project withcookiecutter, appropriate deployment files will be generated under my_fastagency_app/deployment folder. We'll quickly explain what they contain and how to use them.

Registering workflow with a UI or a provider#

The workflow definition will be registered either directly with an UI object or a network provider that will propagate messages back and forth over a network protocol.

In the case of a simple Mesop application, will register the workflow directly with MesopUI object and run it in the same process.

from fastagency import FastAgency
from fastagency.ui.mesop import MesopUI
from fastagency.ui.mesop.auth.basic_auth import BasicAuth

from ..workflow import wf

auth = BasicAuth(
    # TODO: Replace `allowed_users` with the desired usernames and their
    # bcrypt-hashed passwords. One way to generate bcrypt-hashed passwords
    # is by using online tools such as https://bcrypt.online
    # Default password for all users is `password`
    allowed_users={
        "admin": "$2y$10$ZgcGQlsvMoMRmmW4Y.nUVuVHc.vOJsOA7iXAPXWPFy9DX2S7oeTDa",  # nosemgrep: generic.secrets.security.detected-bcrypt-hash.detected-bcrypt-hash
        "user@example.com": "$2y$10$ZgcGQlsvMoMRmmW4Y.nUVuVHc.vOJsOA7iXAPXWPFy9DX2S7oeTDa",  # nosemgrep: generic.secrets.security.detected-bcrypt-hash.detected-bcrypt-hash
    },
)

ui = MesopUI(auth=auth)


app = FastAgency(
    provider=wf,
    ui=ui,
    title="My FastAgency App",
)

# start the fastagency app with the following command
# gunicorn my_fastagency_app.deployment.main:app

In the case of FastAPI application, we will create an FastAPIAdapter and then include a router to the FastAPI application. The adapter will have all REST and Websocket routes for communicating with a client.

from typing import Any

from fastagency.adapters.fastapi import FastAPIAdapter
from fastapi import FastAPI

from ..workflow import wf

adapter = FastAPIAdapter(provider=wf)

app = FastAPI()
app.include_router(adapter.router)


# this is optional, but we would like to see the list of available workflows
@app.get("/")
def list_workflows() -> dict[str, Any]:
    return {"Workflows": {name: wf.get_description(name) for name in wf.names}}


# start the adapter with the following command
# uvicorn my_fastagency_app.deployment.main_1_fastapi:app --reload

In the case of NATS.io application, we will create an NatsAdapter and then add it to a FastAPI application using the lifespan parameter. The adapter will have all REST and Websocket routes for communicating with a client.

import os
from typing import Any

from fastagency.adapters.nats import NatsAdapter
from fastapi import FastAPI

from ..workflow import wf

nats_url = os.environ.get("NATS_URL", "nats://localhost:4222")
user: str = os.environ.get("FASTAGENCY_NATS_USER", "fastagency")
password: str = os.environ.get("FASTAGENCY_NATS_PASSWORD", "fastagency_nats_password")

adapter = NatsAdapter(provider=wf, nats_url=nats_url, user=user, password=password)

app = FastAPI(lifespan=adapter.lifespan)


# this is optional, but we would like to see the list of available workflows
@app.get("/")
def list_workflows() -> dict[str, Any]:
    return {"Workflows": {name: wf.get_description(name) for name in wf.names}}


# start the adapter with the following command
# uvicorn my_fastagency_app.deployment.main_1_nats:app --reload
The NatsAdapter requires a running NATS server. The easiest way to start the NATS server is by using Docker. FastAgency uses the JetStream feature of NATS and also utilizes authentication.

websocket {
    # listen: localhost:9222
    port: 9222
    no_tls: true
    compress: true
}

jetstream {}

accounts {
  AUTH {
    jetstream: enabled
    users: [
      { user: fastagency, password: $FASTAGENCY_NATS_PASSWORD }
    ]
  }
  APP {
    jetstream: enabled
  }
  SYS {}
}

authorization {
  auth_callout {
    issuer: $NATS_PUB_NKEY
    auth_users: [ fastagency ]
    account: AUTH
  }
}

system_account: SYS

In the above NATS configuration, we define a user called fastagency, and its password is read from the environment variable FASTAGENCY_NATS_PASSWORD. We also enable JetStream in NATS and configure NATS to serve via the appropriate ports.

Adapter Chaining#

In case we used network adapters in the step above, we can chain additional network adapters or UI objects to it.

Not applicable for this setup as there are no adapters used.

There is an additional specification file for an application using MesopUI to connect to the FastAPIAdapter

main_2_mesop.py

from fastagency.adapters.fastapi import FastAPIAdapter
from fastagency.app import FastAgency
from fastagency.ui.mesop import MesopUI
from fastagency.ui.mesop.auth.basic_auth import BasicAuth

fastapi_url = "http://localhost:8008"

provider = FastAPIAdapter.create_provider(
    fastapi_url=fastapi_url,
)
auth = BasicAuth(
    # TODO: Replace `allowed_users` with the desired usernames and their
    # bcrypt-hashed passwords. One way to generate bcrypt-hashed passwords
    # is by using online tools such as https://bcrypt.online
    # Default password for all users is `password`
    allowed_users={
        "admin": "$2y$10$ZgcGQlsvMoMRmmW4Y.nUVuVHc.vOJsOA7iXAPXWPFy9DX2S7oeTDa",  # nosemgrep: generic.secrets.security.detected-bcrypt-hash.detected-bcrypt-hash
        "user@example.com": "$2y$10$ZgcGQlsvMoMRmmW4Y.nUVuVHc.vOJsOA7iXAPXWPFy9DX2S7oeTDa",  # nosemgrep: generic.secrets.security.detected-bcrypt-hash.detected-bcrypt-hash
    },
)

ui = MesopUI(auth=auth)


app = FastAgency(
    provider=provider,
    ui=ui,
    title="My FastAgency App",
)

# start the provider with the following command
# gunicorn my_fastagency_app.deployment.main_2_mesop:app -b 0.0.0.0:8888 --reload

Above, we created NATS.io provider that will start brokers waiting to consume initiate workflow messages from the message broker. Now, we create FastAPI service interacting with NATS.io provider:

main_2_fastapi.py

from os import environ

from fastagency.adapters.fastapi import FastAPIAdapter
from fastagency.adapters.nats import NatsAdapter
from fastapi import FastAPI

nats_url = environ.get("NATS_URL", "nats://localhost:4222")
nats_user: str = "fastagency"
nats_password: str = environ.get("FASTAGENCY_NATS_PASSWORD", "fastagency_nats_password")

provider = NatsAdapter.create_provider(
    nats_url=nats_url, user=nats_user, password=nats_password
)

adapter = FastAPIAdapter(
    provider=provider,
)

app = FastAPI()
app.include_router(adapter.router)


# this is optional, but we would like to see the list of available workflows
@app.get("/")
def read_root() -> dict[str, dict[str, str]]:
    return {
        "Workflows": {name: provider.get_description(name) for name in provider.names}
    }


# start the provider with the following command
# uvicorn my_fastagency_app.deployment.main_2_fastapi:app --host 0.0.0.0 --port 8008 --reload

Finally, we create Mesop app communicating with the FastAPI application:

main_3_mesop.py

from fastagency.adapters.fastapi import FastAPIAdapter
from fastagency.app import FastAgency
from fastagency.ui.mesop import MesopUI
from fastagency.ui.mesop.auth.basic_auth import BasicAuth

fastapi_url = "http://localhost:8008"

provider = FastAPIAdapter.create_provider(
    fastapi_url=fastapi_url,
)
auth = BasicAuth(
    # TODO: Replace `allowed_users` with the desired usernames and their
    # bcrypt-hashed passwords. One way to generate bcrypt-hashed passwords
    # is by using online tools such as https://bcrypt.online
    # Default password for all users is `password`
    allowed_users={
        "admin": "$2y$10$ZgcGQlsvMoMRmmW4Y.nUVuVHc.vOJsOA7iXAPXWPFy9DX2S7oeTDa",  # nosemgrep: generic.secrets.security.detected-bcrypt-hash.detected-bcrypt-hash
        "user@example.com": "$2y$10$ZgcGQlsvMoMRmmW4Y.nUVuVHc.vOJsOA7iXAPXWPFy9DX2S7oeTDa",  # nosemgrep: generic.secrets.security.detected-bcrypt-hash.detected-bcrypt-hash
    },
)

ui = MesopUI(auth=auth)


app = FastAgency(
    provider=provider,
    ui=ui,
    title="My FastAgency App",
)

# start the provider with the following command
# gunicorn my_fastagency_app.deployment.main_3_mesop:app -b 0.0.0.0:8888 --reload

Authentication#

FastAgency provides three types of authentication mechanisms: Basic Authentication, Firebase Authentication and no authentication (more will be added soon). The default authentication mechanism is Basic Authentication. You can choose the type of authentication while setting up the project with Cookiecutter.

To use Basic Authentication, configure the desired usernames and their bcrypt hashed passwords in the BasicAuth class and apply the authentication object to the MesopUI object.

main.py

from fastagency import FastAgency
from fastagency.ui.mesop import MesopUI
from fastagency.ui.mesop.auth.basic_auth import BasicAuth

from ..workflow import wf

auth = BasicAuth(
    # TODO: Replace `allowed_users` with the desired usernames and their
    # bcrypt-hashed passwords. One way to generate bcrypt-hashed passwords
    # is by using online tools such as https://bcrypt.online
    # Default password for all users is `password`
    allowed_users={
        "admin": "$2y$10$ZgcGQlsvMoMRmmW4Y.nUVuVHc.vOJsOA7iXAPXWPFy9DX2S7oeTDa",  # nosemgrep: generic.secrets.security.detected-bcrypt-hash.detected-bcrypt-hash
        "user@example.com": "$2y$10$ZgcGQlsvMoMRmmW4Y.nUVuVHc.vOJsOA7iXAPXWPFy9DX2S7oeTDa",  # nosemgrep: generic.secrets.security.detected-bcrypt-hash.detected-bcrypt-hash
    },
)

ui = MesopUI(auth=auth)


app = FastAgency(
    provider=wf,
    ui=ui,
    title="My FastAgency App",
)

# start the fastagency app with the following command
# gunicorn my_fastagency_app.deployment.main:app

main_2_mesop.py

from fastagency.adapters.fastapi import FastAPIAdapter
from fastagency.app import FastAgency
from fastagency.ui.mesop import MesopUI
from fastagency.ui.mesop.auth.basic_auth import BasicAuth

fastapi_url = "http://localhost:8008"

provider = FastAPIAdapter.create_provider(
    fastapi_url=fastapi_url,
)
auth = BasicAuth(
    # TODO: Replace `allowed_users` with the desired usernames and their
    # bcrypt-hashed passwords. One way to generate bcrypt-hashed passwords
    # is by using online tools such as https://bcrypt.online
    # Default password for all users is `password`
    allowed_users={
        "admin": "$2y$10$ZgcGQlsvMoMRmmW4Y.nUVuVHc.vOJsOA7iXAPXWPFy9DX2S7oeTDa",  # nosemgrep: generic.secrets.security.detected-bcrypt-hash.detected-bcrypt-hash
        "user@example.com": "$2y$10$ZgcGQlsvMoMRmmW4Y.nUVuVHc.vOJsOA7iXAPXWPFy9DX2S7oeTDa",  # nosemgrep: generic.secrets.security.detected-bcrypt-hash.detected-bcrypt-hash
    },
)

ui = MesopUI(auth=auth)


app = FastAgency(
    provider=provider,
    ui=ui,
    title="My FastAgency App",
)

# start the provider with the following command
# gunicorn my_fastagency_app.deployment.main_2_mesop:app -b 0.0.0.0:8888 --reload

main_3_mesop.py

from fastagency.adapters.fastapi import FastAPIAdapter
from fastagency.app import FastAgency
from fastagency.ui.mesop import MesopUI
from fastagency.ui.mesop.auth.basic_auth import BasicAuth

fastapi_url = "http://localhost:8008"

provider = FastAPIAdapter.create_provider(
    fastapi_url=fastapi_url,
)
auth = BasicAuth(
    # TODO: Replace `allowed_users` with the desired usernames and their
    # bcrypt-hashed passwords. One way to generate bcrypt-hashed passwords
    # is by using online tools such as https://bcrypt.online
    # Default password for all users is `password`
    allowed_users={
        "admin": "$2y$10$ZgcGQlsvMoMRmmW4Y.nUVuVHc.vOJsOA7iXAPXWPFy9DX2S7oeTDa",  # nosemgrep: generic.secrets.security.detected-bcrypt-hash.detected-bcrypt-hash
        "user@example.com": "$2y$10$ZgcGQlsvMoMRmmW4Y.nUVuVHc.vOJsOA7iXAPXWPFy9DX2S7oeTDa",  # nosemgrep: generic.secrets.security.detected-bcrypt-hash.detected-bcrypt-hash
    },
)

ui = MesopUI(auth=auth)


app = FastAgency(
    provider=provider,
    ui=ui,
    title="My FastAgency App",
)

# start the provider with the following command
# gunicorn my_fastagency_app.deployment.main_3_mesop:app -b 0.0.0.0:8888 --reload

Run Application Locally#

Once everything is set up, you can run your FastAgency application using the following command:

The preferred way to run the Mesop application is using a Python WSGI HTTP server like Gunicorn on Linux and Mac or Waitress on Windows.

  • In the root of your project run:

Terminal

gunicorn my_fastagency_app.deployment.main:app

In this setup, we need to run two commands in separate terminal windows:

  • Start FastAPI application using uvicorn, in the root of your project run:

Terminal 1

uvicorn my_fastagency_app.deployment.main_1_fastapi:app --host 0.0.0.0 --port 8008 --reload
  • Start Mesop web interface using gunicorn, in the root of your project run:

Terminal 2

gunicorn my_fastagency_app.deployment.main_2_mesop:app -b 0.0.0.0:8888 --reload

The NATS docker container is started automatically by Cookiecutter for this setup. In this setup, we need to run three commands in separate terminal windows:

  • Start FastAPI application that provides a conversational workflow, in the root of your project run::

Terminal 1

uvicorn my_fastagency_app.deployment.main_1_nats:app --reload
  • Start FastAPI application integrated with a NATS messaging system, in the root of your project run::

Terminal 2

uvicorn my_fastagency_app.deployment.main_2_fastapi:app --host 0.0.0.0 --port 8008 --reload
  • Start Mesop web interface using gunicorn, in the root of your project run::

Terminal 3

gunicorn my_fastagency_app.deployment.main_3_mesop:app -b 0.0.0.0:8888 --reload

Output#

The outputs will vary based on the interface, here is the output of the last terminal starting UI:

[2024-10-10 13:19:18 +0530] [23635] [INFO] Starting gunicorn 23.0.0
[2024-10-10 13:19:18 +0530] [23635] [INFO] Listening at: http://127.0.0.1:8000 (23635)
[2024-10-10 13:19:18 +0530] [23635] [INFO] Using worker: sync
[2024-10-10 13:19:18 +0530] [23645] [INFO] Booting worker with pid: 23645

Initial message

Deployment#

Building the Docker Image#

If you created the project using Cookiecutter, then building the Docker image is as simple as running the provided script, as shown below:

./scripts/build_docker.sh

Running the above command will build the Docker image. If the build is successful, you will see output similar to the following:

Output
Building fastagency docker image
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.41kB done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/python:3.12
#2 DONE 1.6s

#3 [internal] load .dockerignore
#3 transferring context: 34B done
#3 DONE 0.0s

#4 [internal] load build context
#4 transferring context: 14.06kB done
#4 DONE 0.0s

#5 [1/8] FROM docker.io/library/python:3.12@sha256:fccc38d7080ff9883ee85a65a340384d04eb1c148a7222439b3dc5d4f0f72025
...
...
...
#12 DONE 0.2s

#13 exporting to image
#13 exporting layers
#13 exporting layers 0.9s done
#13 writing image sha256:d5b5432294fa293e3f8d5a2128c2ff012faa640fb552c43ce5faf240bce8bc0f done
#13 naming to docker.io/library/deploy_fastagency done
#13 DONE 0.9s
Successfully built fastagency docker image

Running the Docker Image#

Similarly, running the Docker container is as simple as running the provided script, as shown below:

./scripts/run_docker.sh

Running the above command will start the Docker container in the foreground with the following output:

Output
Number of workers: 1
Nginx config:
upstream mesop_backend {
    # Enable sticky sessions with IP hash
    ip_hash;

    server 127.0.0.1:8889;

}

server {
    listen 8888;
    server_name localhost;

    # Security headers
    add_header X-Frame-Options "SAMEORIGIN";
    add_header X-Content-Type-Options "nosniff";
    add_header X-XSS-Protection "1; mode=block";

    location / {
        proxy_pass http://mesop_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_redirect off;
        proxy_buffering off;

        # WSGI support
        proxy_set_header X-Forwarded-Host $server_name;

        # WebSocket support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}
Starting gunicorn on port 8889
2024/11/12 07:52:47 [warn] 10#10: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:1
[2024-11-12 07:52:47 +0000] [12] [INFO] Starting gunicorn 23.0.0
[2024-11-12 07:52:47 +0000] [12] [INFO] Listening at: http://0.0.0.0:8889 (12)
[2024-11-12 07:52:47 +0000] [12] [INFO] Using worker: sync
[2024-11-12 07:52:47 +0000] [29] [INFO] Booting worker with pid: 29
flaml.automl is not available. Please install flaml[automl] to enable AutoML functionalities.
2024-11-12 07:52:49,054 [INFO] Patching static file serving in Mesop
2024-11-12 07:52:49,055 [INFO] Initializing MesopUI: <fastagency.ui.mesop.mesop.MesopUI object at 0x74b4128986b0>
2024-11-12 07:52:49,059 [INFO] Initialized MesopUI: <fastagency.ui.mesop.mesop.MesopUI object at 0x74b4128986b0>
2024-11-12 07:52:49,059 [INFO] Initializing MesopUI: <fastagency.ui.mesop.mesop.MesopUI object at 0x74b403fffb30>
2024-11-12 07:52:49,059 [INFO] Initialized MesopUI: <fastagency.ui.mesop.mesop.MesopUI object at 0x74b403fffb30>
2024-11-12 07:52:49,059 [INFO] Initializing FastAgency <FastAgency title=Write FastAgency Docs> with workflows: <fastagency.adapters.fastapi.base.FastAPIProvider object at 0x74b416f21010> and UI: <fastagency.ui.mesop.mesop.MesopUI object at 0x74b403fffb30>
2024-11-12 07:52:49,059 [INFO] Initialized FastAgency: <FastAgency title=Write FastAgency Docs>
2024-11-12 07:52:49,068 [INFO] Importing autogen.base.py
INFO:     Started server process [11]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8008 (Press CTRL+C to quit)

Deploying to Fly.io#

If you created the project using Cookiecutter, there are built-in scripts to deploy your workflow to Fly.io. In Fly.io, the application namespace is global, so the application name you chose might already be taken. To check your application's name availability and to reserve it, you can run the following script:

./scripts/register_to_fly_io.sh

Running the above command will prompt you to log in to your Fly.io account (if not already logged in) by opening a fly.io URL in your browser. The login prompt will look like this:

Checking if already logged into fly.io
Logging into fly.io
failed opening browser. Copy the url (https://fly.io/app/auth/cli/78366a6d347a377a6e346465776167726f6b693537666a333674346978626d37) into a browser and continue
Opening https://fly.io/app/auth/cli/78366a6d347a377a6e346465776167726f6b693537666a333674346978626d37 ...

Waiting for session...

After logging in, the script will check if the application name is available and reserve it if it is. The reserved app domain for Fly.io is saved in a file called "registered_app_domain.txt". The output of running the script will look like this:

Output
successfully logged in
Registering app name in fly.io
New app created: test-registration
App name registered successfully
Registered app name is:
test-registration.fly.dev

Deploying to Fly.io manually#

Once you have reserved your application name, you can test whether you can deploy your application to Fly.io using the following script:

./scripts/deploy_to_fly_io.sh

This script will deploy your application to Fly.io without any further input. The output will look like this:

Output
Checking if already logged into fly.io
Already logged into fly.io
Deploying to fly.io
==> Verifying app config
Validating fly.toml
✓ Configuration is valid
--> Verified app config
==> Building image
==> Building image with Depot
--> build:  (​)
[+] Building 15.2s (13/13) FINISHED
...
...
...
--> Build Summary:  (​)
--> Building image done
image: registry.fly.io/write-fastagency-docs-delicate-waterfall-7272:deployment-01JCFQP31QFJWA9HVVYPZYH1QN
image size: 498 MB

Watch your deployment at https://fly.io/apps/write-fastagency-docs-delicate-waterfall-7272/monitoring

Provisioning ips for write-fastagency-docs-delicate-waterfall-7272
Dedicated ipv6: 2a09:8280:1::4f:f553:0
Shared ipv4: 66.241.124.140
Add a dedicated ipv4 with: fly ips allocate-v4

This deployment will:
* create 2 "app" machines

No machines in group app, launching a new machine
Creating a second machine to increase service availability
Finished launching new machines
-------
NOTE: The machines for [app] have services with 'auto_stop_machines = "stop"' that will be stopped when idling

-------
Checking DNS configuration for write-fastagency-docs-delicate-waterfall-7272.fly.dev

Visit your newly deployed app at https://write-fastagency-docs-delicate-waterfall-7272.fly.dev/
Setting secrets
Updating existing machines in 'write-fastagency-docs-delicate-waterfall-7272' with rolling strategy

-------
✔ [1/2] Machine 48e2764ce93e58 [app] update succeeded
✔ [2/2] Machine e825942c739518 [app] update succeeded
-------
Checking DNS configuration for write-fastagency-docs-delicate-waterfall-7272.fly.dev

This is only for testing purposes. You should deploy using GitHub Actions as explained in the next section.

Deploying to Fly.io using GitHub Actions#

Cookiecutter generated all the necessary files to deploy your application to Fly.io using GitHub Actions. Github Actions deployment worfkow will not work unless you follow these steps:

  1. Create a new GitHub repository with your FastAgency project name.

  2. Add the following secrets to your GitHub repository:

    • FLY_API_TOKEN: Your Fly.io API token.
    • OPENAI_API_KEY: Your OpenAI API key.

    To learn how to create keys and add them as secrets, use the following links:

  3. In your devcontainer's terminal, run the following commands to commit and push your project to the new GitHub repository:

    git init
    git add .
    git commit -m "Initial commit"
    git remote add origin https://github.com/<username>/<repo-name>.git
    git branch -M main
    git push -u origin main
    

    Make sure to replace <username> and <repo-name> with your GitHub username and repository name, respectively.

Once these steps are complete, the GitHub Actions workflow will automatically deploy your application to Fly.io. And continue to do so every time you push changes to your repository's main branch.

Deploying to Azure#

If you created the project using Cookiecutter, there are built-in scripts to deploy your workflow to Azure using Azure Container Apps. Please read the following sections to learn how to deploy your application to Azure.

Deploying to Azure manually#

You can test whether you can deploy your application to Azure using the following script:

./scripts/deploy_to_azure.sh

Running the above command will prompt you to log in to your Azure account (if not already logged in) by opening a azure login URL in your browser. The login prompt will look like this:

Retrieving tenants and subscriptions for the selection...

[Tenant and subscription selection]

No     Subscription name                     Subscription ID                       Tenant
-----  ------------------------------------  ------------------------------------  ------------------------
[1] *  Subscription 1                        18a56427-c3d6-4bd8-96fe-c99d96d5f1ef  airt technologies d.o.o.
[2]    Subscription 2                        66699c06-f666-471f-b390-9d6af1f1b522  airt technologies d.o.o.

The default is marked with an *; the default tenant is 'airt technologies d.o.o.' and subscription is 'Subscription 1' (18a56427-c3d6-4bd8-96fe-c99d96d5f1ef).

If you have multiple subscriptions, you will be prompted to select the subscription you want to use. After selecting the subscription, the script will deploy your application to Azure without any further input. The output will look like this:

Output
Creating resource group if it doesn't exists already
...
Creating azure container registry if it doesn't exists already
...
Login Succeeded
Building and pushing docker image to azure container registry
[+] Building 41.0s (13/13) FINISHED                                                                                                                     docker:default
=> [internal] load build definition from Dockerfile                                                                                                              0.0s
=> => transferring dockerfile: 1.42kB                                                                                                                            0.0s
=> [internal] load metadata for docker.io/library/python:3.12                                                                                                    2.6s
=> [internal] load .dockerignore                                                                                                                                 0.0s
=> => transferring context: 34B                                                                                                                                  0.0s
=> [internal] load build context                                                                                                                                 0.0s
=> => transferring context: 16.56kB                                                                                                                              0.0s
=> [1/8] FROM docker.io/library/python:3.12@sha256:949f3c91300ba0a4db28f04797cbff9bd743a7a0f39e570b9e8d9d7a25dd0334                                             11.0s
...
=> [8/8] RUN adduser --disabled-password --gecos '' appuser     && chown -R appuser /app     && chown -R appuser:appuser /etc/nginx/conf.d /var/log/nginx /var/  0.2s
=> exporting to image                                                                                                                                            0.9s
=> => exporting layers                                                                                                                                           0.9s
=> => writing image sha256:760c47963dd21c224c0a582b7b0dbf9baa73436a3909207a775ceb4161a5b6dd                                                                      0.0s
=> => naming to deployazurefastagencyacr.azurecr.io/deploy-azure-fastagency:latest                                                                               0.0s
The push refers to repository [deployazurefastagencyacr.azurecr.io/deploy-azure-fastagency]
...
latest: digest: sha256:fd693c8cd40be2a889bed8c4c3e83957b1c2c46d9ddc8908a6805f789f34ba58 size: 3259
Checking if container app environment exists
Creating vnet for container app environment
...
Creating container app environment
...
Creating container app
No credential was provided to access Azure Container Registry. Trying to look up credentials...
Adding registry password as a secret with name "deployazurefastagencyacrazurecrio-deployazurefastagencyacr"

Container app created. Access your app at https://deploy-azure-fastagency.purplemoss-8d30e9f2.westeurope.azurecontainerapps.io/

"deploy-azure-fastagency.purplemoss-8d30e9f2.westeurope.azurecontainerapps.io"
Updating container app to expose all the service ports
...
Setting up session affinity
...
Fetching your Azure Container App's hosted URL
Your Azure Container App's hosted URL is: https://deploy-azure-fastagency.purplemoss-8d30e9f2.westeurope.azurecontainerapps.io

This is only for testing purposes. You should deploy using GitHub Actions as explained in the next section.

Deploying to Azure using GitHub Actions#

Cookiecutter generated all the necessary files to deploy your application to Azure using GitHub Actions. Github Actions deployment worfkow will not work unless you follow these steps:

  1. Create a new GitHub repository with your FastAgency project name.

  2. Add the following secrets to your GitHub repository:

    • AZURE_CREDENTIALS: Azure service principal credentials in the following format:
      {
          "clientId": "<Client ID>",
          "clientSecret": "<Client Secret>",
          "subscriptionId": "<Subscription ID>",
          "tenantId": "<Tenant ID>"
      }
      
    • OPENAI_API_KEY: Your OpenAI API key.

    To learn how to create keys and add them as secrets, use the following links:

  3. In your devcontainer's terminal, run the following commands to commit and push your project to the new GitHub repository:

    git init
    git add .
    git commit -m "Initial commit"
    git remote add origin https://github.com/<username>/<repo-name>.git
    git branch -M main
    git push -u origin main
    

    Make sure to replace <username> and <repo-name> with your GitHub username and repository name, respectively.

Once these steps are complete, the GitHub Actions workflow will automatically deploy your application to Azure using Azure Container Apps. And continue to do so every time you push changes to your repository's main branch.