Senior Data Engineer

Beschreibung

Job Zusammenfassung In diesem Job entwickelst du leistungsstarke Datenpipelines, designst passende Daten-Schemas und optimierst die Integration von Daten aus verschiedenen Quellen, um die Qualität für KI-gestützte Systeme sicherzustellen. Job Zusammenfassung In diesem Job entwickelst du leistungsstarke Datenpipelines, designst passende Daten-Schemas und optimierst die Integration von Daten aus verschiedenen Quellen, um die Qualität für KI-gestützte Systeme sicherzustellen. Deine Rolle im Team Build and own the data pipelines and knowledge systems that power AI automation. Design and own canonical data schemas that structure knowledge for AI consumption - versioned, documented, and treated as engineering artefacts. Evolve schemas as AI system requirements change: assess downstream impact and coordinate with the engineering team before any rollout. Identify and connect to source systems - databases, repositories, APIs, workflow outputs, domain expert inputs - and build reliable, monitored ingestion pipelines for each. Build and maintain production-grade data pipelines using pipeline orchestration tooling (such as Apache Airflow, Prefect, or equivalent): extraction, transformation, validation, loading, error handling, and full observability. Transform raw source material into structured, consistent, agent-consumable knowledge: normalise formats, resolve conflicts, enforce schema conformance, and eliminate ambiguity. Maintain all data pipelines and knowledge systems in production: monitor health, manage dependencies, handle source system changes, and ensure nothing degrades silently over time. Work in a closed loop with the engineering team: diagnose knowledge gaps when AI outputs fail, fix at the pipeline or schema level, and validate through the evaluation framework. Select and manage data storage infrastructure appropriate to each knowledge type - relational databases, document stores, and vector databases (such as Pinecone, Weaviate, or pgvector) - with AI consumption as the deciding factor. Translate between domain and engineering language in both directions - extracting knowledge requirements from non-technical stakeholders and communicating data decisions and tradeoffs back clearly. Unsere Erwartungen an dich Qualifikationen Proven track record building and owning production data pipelines end-to-end: source integration, transformation, validation, loading, observability, and ongoing maintenance. Strong data schema design skills: canonical data modelling, schema evolution management, downstream impact assessment, and decision documentation. Data cleaning and transformation expertise: normalisation, conflict resolution, format standardisation, and quality validation at scale using data transformation tools (such as dbt or equivalent). Working knowledge of how RAG systems and context assembly operate - sufficient to make data design decisions with AI consumption as the primary constraint. Ability to interrogate and profile data across sources - querying, inspecting, and validating content using SQL and Python to identify anomalies, gaps, and quality issues before they reach AI systems. Erfahrung Experience integrating with diverse source systems - APIs, databases, document repositories, event streams - with reliable, monitored ingestion pipelines. Hands-on experience with pipeline orchestration tooling (such as Apache Airflow, Prefect, or equivalent): scheduling, dependency management, failure handling, and pipeline observability. Experience with vector databases (such as Pinecone, Weaviate, or pgvector) and relational or document stores: schema design, indexing, and production data management. Experience partnering with AI or ML engineers: translating knowledge requirements into pipeline tasks and closing feedback loops between AI output quality and data quality. Cloud platform experience (AWS, Azure, or GCP): deploying and operating data pipelines and storage systems at production reliability standards. Unser Angebot Please submit your resume and a brief cover letter outlining your experience and interest in the role. Atari is an equal opportunity employer and we are committed to providing a workplace free from harassment and discrimination. Themen mit denen du dich im Job beschäftigst AI Data Economy Metadaten Level: Senior Job Feld: Data, Back End Anstellung: Vollzeit Vertragsart: Unbefristetes Dienstverhältnis Arbeitsmodell: Onsite Unternehmenstyp: Etablierte Firma Branche: Medien, Verlagswesen Ort: Münster (Hessen) | 64839 | Deutschland

Tech Stack