Job Description

Data Engineering , includes tasks like building data pipelines, ensuring data quality, optimizing data models. The role requires expertise in Azure services like Azure Data Lake, SQL, Cosmos DB, Data Factory, Synapse Analytics, Databricks .

You will be responsible for designing, developing, and leading data engineering initiatives with lakehouse architecture.

You will work closely with cross-functional teams to understand business requirements and translate them into technical solutions.

This role requires strong leadership and communication skills and mentoring data engineering teams.

Key Responsibilities:

• Design, configure, and optimize data architectures and pipelines to meet business needs.
• Develop and embed data engineering pipelines as part of multi tenant B2B SaaS product.
• Work closely with cross-functional teams, to understand business requirements and translate them into technical solutions.
• Migrating existing data to new data model, including logging, reconciliation, data quality and error handling.

Required Skills and Experience:

• Extensive experience in data engineering, including data warehousing, data lakes, and lakehouses.
• Lead the design and implementation of data pipelines for ingesting, transforming, and loading data into Azure data & analytics services for structured and semi structured data.
• Experience in developing data services and pipelines with API integration for multitenant application.
• Develop best practices including reusable code, libraries, patterns, and consumable frameworks for cloud-based data warehousing and ETL.
• Design and implement efficient data models and database schemas for structured and unstructured data, optimizing storage and access for performance and scalability.
• Strong analytical and problem-solving skills, with the ability to interpret complex data sets and derive meaningful insights.
• Extensive experience with Azure data & analytics services, including ADLS, Azure SQL, Cosmos DB, Data Factory, Synapse Analytics, Azure Functions, Azure App Service, Apache Spark, Python, Databricks, and other related services.
• Strong understanding and experience designing and development of data warehousing, data modeling, data integration, and ETL/ELT processes.
• Proficiency in scripting languages like Python, SQL, and/or other languages relevant to data engineering tasks.
• Monitor and Optimize data pipelines and data models for performance, scalability, and cost-effectiveness, identify bottlenecks, and implement optimization strategies.
• Understanding of data governance principles, data security best practices, and compliance requirements.
• Experience with data modeling and database design principles.
• Familiarity with data governance and security best practices.
• Provide advanced troubleshooting and resolution of data pipelines and operations issues.
• Experience with relational databases and data warehouse technologies.

Leadership and Communication:

• Proven leadership skills, with the ability to lead, mentor, and motivate a team.
• Oversee a team of data engineers, providing technical guidance, mentorship, and support.
• Provide guidance and support to team members, lead, mentor, and guide data engineering teams, fostering a collaborative and high-performing environment.

Documentation and Training:

• Develop and maintain documentation, including design documents, data pipelines with governance, Standard Operating Procedures (SOPs) and training materials for the team.
• Excellent written and verbal communication skills, with the ability to effectively communicate technical information to both technical and non-technical audiences.
• Experience with Agile Methodologies:
• Familiarity with Agile project management methodologies. "

Job Tags

Similar Jobs