Senior Software Engineer, Spark Platform

The posting, as DoorDash wrote it

About the Team

The Spark Platform team owns and operates DoorDash's Apache Spark ecosystem — the execution runtime, remote shuffle service, cluster scheduler, and reliability tooling that powers the company's data, analytics, and ML workloads. We run Spark across the company at significant scale and continue to expand the workloads, capabilities, and consumer base we serve. Orchestrating and operating thousands of Spark cluster deployments is a complex distributed system problem which the team invests heavily in runtime optimization, systems architecture, multi-tenant scheduling, and end-user tooling.

About the Role

As a Senior Software Engineer on Spark Platform, you will set the technical direction for our in-house Spark deployment and shape the architecture that will run DoorDash's data, analytics, and ML compute for the next five years and beyond. You will own the deep, cross-cutting problems that span the runtime, the shuffle service, the scheduler, and the overall service reliability — making the architectural calls that compound across the platform's lifetime. You will partner with the Engineering Manager on technical roadmap, hiring, and team shape, and act as the senior technical voice in cross-team partnerships with Data Engineering, ML Platform, and product engineering teams that depend on the platform.

You must be located in San Francisco, Sunnyvale, Seattle, or New York City for this hybrid position. You will report into the Engineering Manager on our Spark Platform team.

You're excited about this opportunity because you will…

• Set the multi-year technical direction for an in-house Spark-on-Kubernetes platform — runtime, shuffle, scheduler, reliability — and make the architectural calls that compound for years.
• Own the deepest distributed-systems problems on the team: shuffle architecture, multi-tenant scheduling, runtime performance, and the failure modes that only show up at scale.
• Partner with the Engineering Manager on technical roadmap, hiring, interview design, and team shape as the team continues to grow.
• Uplevel the rest of the team through design reviews, mentorship, and raising the bar on what we ship.
• Represent Spark Platform in cross-team architecture forums and shape how data, analytics, and ML workloads land on the platform.

We're excited about you because…

• B.S., M.S., or PhD in Computer Science or equivalent.
• 6+ years of industry experience designing and operating distributed systems at scale.
• Deep, hands-on experience with Apache Spark — internals, query execution, shuffle, the executor/driver model — at platform scale on Amazon EMR, Databricks, or an in-house deployment, with a focus on platform operations (runtime upgrades, cluster lifecycle, shuffle, observability, multi-tenant scheduling) rather than authoring individual Spark jobs.
• Production experience with one or more of: remote/external shuffle systems (Celeborn, Magnet, Cosco, or similar), batch/big-data schedulers (YuniKo

[…]
success have no place at DoorDash. We value a diverse workforce – people who identify as women, non-binary or gender non-conforming, LGBTQIA+, American Indian or Native Alaskan, Black or African American, Hispanic or Latinx, Native Hawaiian or Other Pacific Islander, differently-abled, caretakers and parents, and veterans are strongly encouraged to apply. Thank you to the Level Playing Field Institute for this statement of non-discrimination.

Pursuant to the San Francisco Fair Chance Ordinance, Los Angeles Fair Chance Initiative for Hiring Ordinance, and any other state or local hiring regulations, we will consider for employment any qualified applicant, including those with arrest and conviction records, in a manner consistent with the applicable regulation.

If you need any accommodations, please inform your recruiting contact upon initial connection.

Notice Regarding Use of AI and Automated Tools: To streamline our hiring process, DoorDash utilizes an automated recruitment tool called Metaview.

How it works: Metaview assists our recruiting team by evaluating job related qualifications and characteristics in connection with hiring. The tool is designed and used to support - rather than replace - human decision-making; trained personnel make final decisions with meaningful human review and oversight, and DoorDash does not use Metaview or other AI-enabled tool in a manner that has the effect of subjecting applicants or employees to discrimination based on any protected characteristic or proxy or for engaging in any protected activity under applicable law.

Data Retention, Privacy & Bias Audit: Data collected during this process is retained in accordance with our Candidate Privacy Policy https://help.doordash.com/en-us/legal/article/ax-privacy-notice and applicable state laws. In compliance with New York City Local Law 144, the independent bias audit summary for Metaview is publicly available for review at https://trust.warden-ai.com/metaview/application-review .