AI Evals Engineer — Evaluation Datasets & Ground Truth

The posting, as Prophetic Technologies Inc wrote it

Prophetic Software is not able to sponsor employment visas now or in the future. Candidates must be authorized to work in the United States without current or future sponsorship to be considered for this role.

About Prophetic

Real estate development is a multi-billion-dollar industry that has run on fragmented data, manual processes, and gut instinct for decades. Prophetic is changing that. We're building the AI-native platform that enables homebuilders, developers, and investors to find, analyze, and act on land opportunities from a single system — powered by proprietary technologies that process billions of data points across all 50 states. We are the market leader in our space, and our customers don't just use the product — they love it. We're not making teams more efficient. We're changing how they operate.

Prophetic is scaling fast — demand is outpacing our ability to hire, and we're just getting started. This is a once-in-a-lifetime team: sharp, low ego, deeply collaborative, and obsessed with building the best product in the industry. Come disrupt an industry with us.

WHY THIS ROLE EXISTS

We iterate on pipeline components constantly: prompts, models, classifier thresholds, retrieval logic, harness design. We are hiring a full-time engineer to produce, maintain, and defend the evaluation datasets that let every meaningful component in our stack be measured. You will own ground truth at Prophetic.

This is not an evals-infrastructure role and it is not a labeling-operations role, although you’ll touch both. Your deliverable is trusted data: for a given module, a versioned set of inputs and expected outputs, plus a written definition of what “correct” means and how confident we should be in the labels.

WHAT YOU’LL DO

Decide what needs to be measured, and how

• Read our pipelines and system architecture, sit with product and engineering, and decompose each system into evaluable modules with explicit input → expected-output contracts.
• For each module, define what “correct” means in writing: rubrics, label schemas, edge-case policies, and the tolerances that matter to the business.
• Prioritize. We have more modules than you can cover in year one; you’ll decide where a validation set unblocks the most iteration, with minimal direction from principal engineers.

Source the data by whatever means is cheapest and most trustworthy for that module

• Production sampling: pull stratified, de-identified samples from real traffic so eval sets reflect what the system actually sees, including the long tail.
• Human labeling: scope and run labeling programs, write annotation guidelines, build calibration sets, measure inter-annotator agreement, and manage vendors (including offshore labeling teams) or internal subject-matter experts. You own label quality, not just label throughput.
• Synthetic / oracle-generated ground truth: where a task is tractable for a frontier model given enough compute (long context, multi-pass, to

[…]
ity. Remote, hybrid, or in-office with a great Portland, OR headquarters.

• Self-starters who thrive in ambiguity. We're a startup. If you need someone to tell you what to do every morning, this isn't the place; if you come alive solving hard problems with smart people, it is.

Benefits (US-based Full-time)

• 100% medical, dental & vision insurance coverage for you; 30% coverage for dependents
• Competitive salary and meaningful early-stage equity
• Unlimited PTO
• Hybrid/Remote stipend
• In-office perks: snacks, drinks, coffee, ping-pong table, and more — plus cuddles from Olive, our in-office Doberman!
• Budget for intra-office travel
• 2–3 annual team meetups in person

How We Work

These four themes are how we think, build, and lead:

• Truth — Say the hard thing. We speak plainly, flag problems early, and share what's actually going on.
• Precision — Sign your name to it. Our customers make million-dollar decisions on our data. We get the details right.
• Velocity — Speed compounds. Waiting doesn't. We ship daily and launch features every two to three weeks.
• Ownership — Own it, then solve it. We own outcomes, not tasks. If something's broken, we fix it.

EEO

Prophetic Software is an equal opportunity employer. We are committed to creating an inclusive environment for all employees and applicants, and we make employment decisions without regard to race, color, religion, sex (including pregnancy, childbirth, or related medical conditions), gender identity or expression, sexual orientation, national origin, age, disability, genetic information, veteran or military status, marital status, or any other characteristic protected under applicable federal, state, or local law. We actively seek to build a diverse team and encourage candidates from underrepresented backgrounds to apply. Reasonable accommodations are available upon request for qualified individuals with disabilities throughout the application and employment process.