Data Engineer

Job Description

Required Skill Set: Programming & Data Engineering: Python (PySpark, Pandas, NumPy), SQL(T-SQL, PostgreSQL, MySQL) Apache Spark, Data Cleaning. ETL/ELT Tools: ADF Data Flows, Databricks Notebooks, OD. Snowflake, Microsoft Fabric. SQL Pools, AWS. CloudSQL (Postgres), Oracle Data Mart, MySQL, MSSQL

Key Responsibilities

• Develop and optimise complex SQL queries to manage large-scale data processing efficiently.
• Design, build, and maintain database objects such as tables, indexes, and views to support reporting and analytics.
• Employ advanced querying techniques, including joins and temporary tables, to transform and manipulate data. Fine-tune query performance and proactively troubleshoot bottlenecks to ensure optimal data retrieval.
• Implement data security practices including role-based access control (RBAC), row-level security (RLS), and data masking.
• Integrate data pipelines with downstream systems including data warehouses, APIs, and BI tools such as Power BI or Tableau.
• Architect, deploy, and maintain end-to-end data pipelines using Azure Data Factory.
• Configure linked services and datasets within ADF, ensuring seamless data ingestion via Copy Activity and API integrations.
• Orchestrate data workflows, set triggers, and monitor pipeline health, with proactive alerting for failures. Design, develop, and manage ETL processes leveraging SSIS to streamline data integration workflows.
• Optimize ETL packages for scalability and performance, handling debugging and error management.
• Create and maintain comprehensive reports and dashboards through SSRS and Power BI. Ensure visualizations are accurate, insightful, and support business decision-making.
• Support data quality, security, and compliance standards across all reporting and processing activities.

Essential Skills & Experience

• Demonstrable expertise in writing high-performance SQL queries, including indexing, joins, and performance tuning.
• Proven experience with Azure Data Factory for building scalable data pipelines. Strong familiarity with reporting tools such as Power BI and SSRS.
• Hands-on skills with SSIS for developing efficient ETL workflows and Contribute to CI/CD pipeline development and deployment processes for data engineering workflows.
• Sound understanding of cloud-based data architectures and best practices. Knowledge of Microsoft Fabric and semantic modelling principles.
• Solid experience in data modelling, including database object design and optimisation.
• Ability to troubleshoot and resolve complex performance issues effectively.
• Perform data quality checks, validation, and reconciliation to ensure accuracy and consistency of data.
• Excellent organisational skills and attention to detail.