Softura - Careers

We are looking for a highly skilled Software Engineer with strong hands-on experience in Azure Data Factory (ADF) for end-to-end data migration from Azure SQL Server to Azure Cosmos DB. The candidate will be responsible for designing, building, optimizing, and managing data integration and transformation pipelines in Azure.

• Design, develop, and deploy ADF pipelines for migrating structured data from SQL Server to Azure Cosmos DB.
• Create ADF linked services, datasets, and data flows to connect and transform data between heterogeneous sources and targets.
• Develop data ingestion, transformation, and orchestration logic using ADF’s Mapping Data Flows and Wrangling Data Flows.
• Implement parameterization, expressions, variables, and pipeline dependencies for dynamic and reusable pipelines.
• Use control flow activities (e.g., Lookup, For Each, If Condition, Execute Pipeline, Web Activity, Stored Procedure Activity) to orchestrate complex workflows.
• Design and manage trigger-based executions (schedule, tumbling window, event-based).
• Implement incremental loads and change data capture (CDC) strategies to handle delta data migration.
• Integrate ADF pipelines with Azure Key Vault, Azure Data Lake Storage (Gen1/Gen2), Azure Blob Storage, and Cosmos DB.
• Work with Cosmos DB Core/SQL API, designing JSON document models, partitioning strategies, and indexing for optimized migration and query performance.
• Use Azure Monitor and ADF Monitoring Hub for pipeline logging, alerting, and error handling.
• Develop and maintain pipeline execution logs, alerts, and retry mechanisms for failure recovery.
• Collaborate with Azure DevOps teams to manage CI/CD for ADF pipelines using Git integration and ARM templates.
• Ensure data quality, consistency, and completeness post-migration through validation and reconciliation processes.
• Deep understanding of relational database design (normalization, referential integrity, primary/foreign keys).
• Experience using temporary tables, CTEs, and window functions for transformation logic.

Pipeline Development & Orchestration

• Creating and managing pipelines, data flows, and activities.
• Understanding of control flow and data flow

Integration Runtimes (IR)

• Setup and configuration of Self-hosted IR, Azure IR, and Managed VNet IR for hybrid connectivity.

Linked Services & Datasets

• Configuring linked services for SQL Server, Azure SQL, Cosmos DB, Blob Storage, and Data Lake.
• Building datasets for both structured and unstructured data sources.

Mapping Data Flows

• Implementing data transformations (Join, Derived Column, Aggregate, Filter, Conditional Split, Lookup, Sink).
• Performing schema drift handling and type conversions.

Parameterization & Reusability

• Using parameters, variables, and expressions to build dynamic, reusable pipelines.

Triggers & Scheduling

• Creating scheduled, tumbling window, and event-based

Error Handling & Logging

• Implementing robust try-catch, custom error messages, and pipeline failure alerts.

Monitoring & Troubleshooting

• Using ADF Monitoring Hub, Azure Monitor, and Log Analytics for pipeline performance insights.

Performance Tuning

• Optimizing pipeline performance via batch sizes, parallelization, and data partitioning.

Security & Governance

• Integrating with Azure Key Vault for credential management.
• Managing RBAC, Managed Identities, and secure data access

Version Control & CI/CD

• Managing ADF with GitHub or Azure DevOps Git Repos.
• Deploying pipelines via ARM templates and Azure DevOps release pipelines.

Integration with Azure Ecosystem

• Working with Azure Blob Storage, Azure Data Lake, Cosmos DB, Azure SQL Database, and Synapse Analytics.

Advanced Topics

• Handling CDC (Change Data Capture) and incremental loading from SQL Server.
• Data validation and reconciliation post-migration.
• Using Web Activity and REST API integration for external service calls.

• 4–8 years of data engineering experience, with at least 2+ years of hands-on ADF
• Strong in SQL Server: T-SQL, stored procedures, schema design, and query optimization.
• Experience with Azure Cosmos DB (Core/SQL API preferred).
• Understanding of NoSQL data modelling and JSON-based document databases.

Hands-on experience with Azure Storage Services, Key Vault, and Azure DevOps.