Applied AI Engineer

The posting, as Monte Carlo wrote it

About Monte Carlo

Monte Carlo is the agent trust platform that autonomously observes, fixes, and continuously improves agents and the data that powers them — so enterprises can scale trust at the speed of AI. Bayes, our AI teammate, does the work: detecting, triaging, troubleshooting, and fixing issues in AI agents and the data powering them. The result is self-improving agents that get better with every fix instead of drifting. More than 400 enterprises, including Amazon, PepsiCo, CNN, Nasdaq, and JetBlue, rely on Monte Carlo. Learn more at montecarlo.ai https://montecarlo.ai.

THE ROLE

We're building the products that tell enterprises whether their AI agents can be trusted — and we need someone who works end to end, from an ambiguous problem statement through research, prototyping, and production. You'd get the problem, not the spec: research the approaches, prototype, prove what works, build it, and integrate it into the platform alongside our engineering and data science teams. This role exists because agent observability moved from roadmap to revenue faster than anyone predicted, and the work is now on the critical path.

WHAT YOU'LL DO

• Take an open problem end-to-end — from research and prototyping through production, killing what doesn't work before it becomes someone's roadmap
• Design and ship agent-powered features — root-cause analysis, incident triage, monitor generation — and integrate them into the platform with our engineering team
• Build the eval infrastructure that makes those features safe to change: golden datasets, regression suites, offline and online scoring, and the judgment calls about what "good" means
• Own retrieval and context pipelines over customer metadata, lineage, and query history, and instrument agent behavior in production — traces, failure taxonomies, cost and latency budgets — to close the loop on quality
• Partner with data science on detection quality and experiment design, and with PM on what an agent should do versus what it merely can do
• Set the technical bar for how we build with LLMs — patterns, guardrails, and the internal tooling other engineers reuse

WHAT WE'RE LOOKING FOR

• You've built agents in production with real users. Not integrated a framework. Not worked on a team that had one. Built them — agents with real autonomy and internal loops, where the model uses tools and decides what to do next without a human in the middle, and you kept them running once real users showed up. RAG with a wrapper doesn't count. Neither does a set of MCP tools point at an API.
• You've run evals and monitored agents after launch. Agents are non-deterministic, so normal tests don't work on them. You've owned an eval framework — golden datasets, regression suites, offline and online scoring — not a folder of one-off scripts. And you've watched agents in production, not just in dev.
• Python, plus an ML or data science background. Python is your daily language and you're solid

[…]
committed to creating an inclusive environment for all employees.

We are proud to be recognized for our world-class employee experience:

Monte Carlo Named 2025 Databricks Data Governance Partner of the Year https://www.montecarlodata.com/blog-2025-databricks-data-governance-partner-of-the-year/?utm_source=chatgpt.com

We were recently recognized as the #1 Data Observability Platform by G2 for the 4th consecutive quarter. See our G2 reviews here! https://www.g2.com/reports/grid-report-for-data-observability-spring-2025.embed?featured=monte-carlo&secure%5Bgated_consumer%5D=7d02ec0a-326a-40fa-8a44-fab49f67c5f1&secure%5Btoken%5D=6b3c29d18ea50ae0005295b5c63994f97c01cae81bbd3f9ea6abff73c40fde51&utm_campaign=gate-2063400

Monte Carlo Named to G2's Best Software Products of 2026 https://www.montecarlodata.com/blog-monte-carlo-g2-best-software-product-of-2026/

Monte Carlo was featured on Database Trends and Applications (DBTA’s) Trend-Setting Products for 2025! https://www.dbta.com/Editorial/Trends-and-Applications/Trend-Setting-Products-in-Data-and-Information-Management-for-2025-167115.aspx

We are super proud to be named the 2026 Best Place to Work by Built In! https://builtin.com/awards/us/2026/best-places-to-work

Beware of Imposter Recruiters and Job Scams

• All official communication from our recruiting team will come from an @montecarlodata.com http://montecarlodata.com email address.
• We will never ask candidates to provide sensitive personal information (such as bank details, social security numbers, or payment) at any stage of the recruitment process.
• We will never request payment for equipment, training, or application processing.
• Our open positions are always listed on our official careers page: https://jobs.ashbyhq.com/montecarlodata.

If you are contacted by someone claiming to represent Monte Carlo but you’re unsure of their legitimacy, please reach out to us directly at recruiting@montecarlodata.com before sharing any personal information.