Technical Articles for Azure Data Platform

Incremental Data Ingestion using Azure Databricks Auto Loader

There are many ways to ingest the data in standard file formats from cloud storage to Delta Lake, but is there a way to ingest data from 100s of files within few seconds as soon as it lands in a storage account folder? I have recently come across a Customer with requirements to ingest ~1500 […]

Bulk Delete Documents in Azure Cosmos DB Container using Azure Data Factory

If you are already working on building an Azure Data Engineering solution using Azure Data Factory as an orchestration tool and Azure Cosmos DB in a scenario where you may have to delete documents from a particular SQL container programmatically, then you might have already figured out that there is no easy way to do […]

Unzip 7z files using Azure Automation Runbooks and Azure Data Factory

I have recently come across a Customer who is migrating On-prem DW workloads to Azure cloud (using Azure Synapse Analytics, Data Factory, Azure Storage, Power BI) and has a scenario where they have a bunch of 7z compressed data files that are being unzipped using a utility installed in a VM for further processing in […]

Follow My Blog

Get new content delivered directly to your inbox.