Model Madness: A Tournament of Tool Calling
The NCAA Tournament is upon us, and with it comes millions of brackets. When the question of whether anyone wanted to do March Madness came up in Alephic Slack, the conversation quickly shifted to setting up a model bracket to pit all these different AIs against each other. The idea then turned to some execution details, and off we went.
tl;dr: We built a bracket challenge to see which model could best navigate tools to make March Madness picks. The site is at model-madness.alephic.ai
The basic concept was to see which model would win in a basic bracket competition with the aid of some simple tools like web search and fetch. The end result, for some interesting reasons, turned out to be a much more complicated system in which 45 models combined entries across three categories: easy, medium, and hard. In the end, I think the experiment tells an interesting story about what it takes to actually build agents and the wide range of capabilities models have at effectively wielding the tools required for agentic work.
The Basic Setup
For posterity's sake, here are the big pieces of the stack we used to build this:
- NextJS running on Vercel
- Vercel AI SDK for calls and basic AI functionality
- Vercel WorkflowDevKit for asynchronous processing
- Vercel AI gateway for easy access to all 45 models through one provider
- Firecrawl for web search and scraping
The Build
As I said, things started out fairly simply: let's build one basic prompt and set of tools that all the models will use to generate their bracket entry. I decided it was probably best to use a dumb model to test with, since that would give me a good baseline for how much scaffolding was needed to make for successful entries. I decided to go with GPT-4o mini, which is not only old but was designed to be a cheaper/dumber alternative.
