Side Projects

AIBench Arena: Benchmark AI Agents on Your Codebase

Benchmark AI coding agents (Claude Code, Codex, Gemini) against YOUR repo's git history. Public benchmarks test them on someone else's code — AIBench Arena tests them on yours.

AIBench Arena solves a fundamental problem in AI agent evaluation: public benchmarks test agents on unfamiliar codebases, but you need to know how they perform on YOUR code, with YOUR conventions, YOUR tech stack, and YOUR problem patterns.

The Problem with Public Benchmarks

Existing AI coding benchmarks (HumanEval, SWE-bench) test agents on generic problems or open-source repos they may have seen during training. This doesn't answer the question every developer and team actually cares about: "Will this agent help ME write better code faster?"

How AIBench Arena Works

1Point to your GitHub repo or local git repository

2AIBench extracts your actual commit history and code changes

3For each commit, it creates a benchmark task: given the code before, can the AI agent produce the change you made?

4Run multiple AI agents (Claude Code, GitHub Copilot, Gemini Code Assist) on the same tasks

5Compare agents on accuracy, speed, and cost across YOUR codebase

Key Features

Git History Analysis: Automatically parse commit diffs into benchmark tasks

Multi-Agent Support: Test Claude, Codex, Gemini, and custom agents

Diff-based Scoring: Measure how close agent output matches actual committed changes

Cost Tracking: Compare not just accuracy but cost per successful task

Custom Filters: Benchmark specific file types, authors, or time periods

Use Cases

Team Evaluation: Before adopting an AI coding assistant, test it on your team's actual work to see ROI

Agent Selection: Compare multiple AI agents on the same codebase to make data-driven purchasing decisions

Performance Tracking: Benchmark agents over time as they improve to decide when to upgrade

Domain Specificity: See which agents perform best on your domain (web, data, infra, etc.)

Technical Architecture

Git Parser: Extract commit history, diffs, and file metadata

Task Generator: Convert commits into standardized benchmark tasks

Agent Runner: Execute tasks across multiple AI coding assistants

Diff Scorer: Measure similarity between agent output and actual commits

Results Dashboard: Visualize accuracy, speed, and cost metrics

View all projects