Research Engineer - Evals
You'll build the evaluation systems that tell us whether Firecrawl actually works. That sounds simple. It isn't. Our core promise, convert any URL into clean, structured, LLM-ready data reliably, is hard to measure rigorously across millions of different websites, formats, and edge cases. As the systems we're measuring get more complex, the question "did that work?" gets harder, not easier.
This isn't an eval role where you inherit a framework and run benchmarks. You'll design the metrics, build the pipelines, generate the datasets, and own the feedback loop from output quality back to model and product decisions. If you care about what "good" actually means and have the engineering depth to measure it, this is the role.
Salary Range: $250,000–$290,000 USD/year (SF) / $210,000–$224,000 CAD/year (Toronto)
Equity Range: Competitive equity. Details shared during the process.
Location
San Francisco, CA (SF HQ) or Toronto, ON (Toronto Hub). Hybrid, onsite 3+ days a week.
Job Type: Full-Time
Experience: 4+ years in ML, research engineering, or data-heavy backend, with real evaluation work
Work Authorization: Open to candidates already authorized to work in the US or Canada. In the US, we also support H-1B transfers, F-1 OPT and STEM OPT, TN (Canada and Mexico), and E-3 (Australia). We’re not able to sponsor new H-1B visas for candidates outside the US. For Canada, we’ll consider work permit sponsorship case by case through our Toronto Hub.
What You'll Do
Design the metrics that define what "good output" actually means across millions of sites, formats, and edge cases
Build the pipelines and harnesses that measure quality rigorously and at scale
Generate and curate the datasets that make evaluation trustworthy
Own the feedback loop from output quality back to model and product decisions
Turn "did that work?" into an answer the whole team can act on
What We're Looking For
You have the engineering depth to build real evaluation systems, not just run existing ones
You care deeply about what "good" means and how to measure it rigorously
You're comfortable owning ambiguous problems where the metric itself has to be invented
You move fast and close the loop. You'd rather ship, measure, and iterate than perfect on paper
What We're Not Looking For
Someone who only wants to run benchmarks someone else designed
A pure researcher who won't build the systems, or a pure engineer who won't think about methodology
Someone who needs a fully-specced ticket to start