Azure Data Factory v2: Hands-on overview

Whether you're shifting ETL workloads to the cloud or visually building data transformation pipelines, version 2 of Azure Data Factory lets you leverage conventional and open source technologies to move, prep and integrate your data.

The second major version of Azure Data Factory, Microsoft's cloud service for ETL (Extract, Transform and Load), data prep and data movement, was released to general availability (GA)about two months ago. Cloud GAs come so fast and furious these days that it's easy to be jaded. But data integration is too important to overlook, and I wanted to examine the product more closely.

Azure Data Factory v2 allows visual construction of data pipelines

Roughly thirteen years after its initial release, SQL Server Integration Services (SSIS) is still Microsoft's on-premises state of the art in ETL. It's old, and it's got tranches of incremental improvements in it that sometimes feel like layers of paint in a rental apartment. But it gets the job done, and reliably so.

In the cloud, meanwhile, Microsoft's first release of Azure Data Factory (ADF) was, to put it charitably, a minimal viable product (MVP) release. Geared mostly to Azure data services and heavily reliant on a JSON-based, handed-coded approach that only a Microsoft Program Manager could love, ADF was hardly a worthy successor to the SSIS legacy.

Clean slate

But ADF v2 is a whole new product:

• It's visual (the JSON-encoded assets are still there, but that's largely hidden from the user)
• It has broad connectivity across data sources and destinations, of both Microsoft and non-Microsoft pedigrees
• It's modern, able to use Hadoop (including MapReduce, Pig and Hive) and Spark to process the data or use its own simple activity construct to copy data
• It doesn't cut ties with the past; in fact, it serves as a cloud-based environment for running packages designed in with the on-premises SSIS

The four points above tell the high-level story fairly well, but let's drill down a bit to make those points a bit more concrete.

Connectivity galore

First off, data sources. Microsoft of course supports its own products and services. Whether it's Microsoft's on-premises flagship SQL Server, PC thorn-in-the-side Access, or in-cloud options like Azure SQL Database, SQL Data Warehouse, Cosmos DB, Blob, file and table storage or Data Lake Store (v1 or v2), ADF can connect to it. And while a connector for Excel files is conspicuously absent, saving a CSV file from Excel allows its data to be processed in ADF using the File System connector.

But there's lots more, including Oracle, DB2, Sybase and Postgres in the RDBMS world; Teradata, Greenplum, Vertica and Netezza data warehouse platforms; MongoDB, Cassandra and Couchbase from the NoSQL scene; HDFS, Hive, Impala, HBase, Drill, Presto and Spark from the open source realm; SAP BW and SAP HANA in the BI/analytics world; Dynamics, Salesforce, Marketo, Service Now and Zoho from the SaaS world and even Amazon Redshift, Amazon S3, Amazon Marketplace Web Service and Google BigQuery on the competitive cloud side.

ADF v2 connects to an array of data stores, both from the Microsoft and competitive worlds

For processing the data, ADF v2 can use Azure Batch, Data Lake Analytics (U-SQL), HDInsight, Databricks or Machine Learning Services. And while all of these are Azure services, they deliver a variety of open source technologies within them.

Structured activities

The core unit of work in ADF is a pipeline, made up of a visual flow of individual activities. Activities are little building blocks, providing flow and data processing functionality. What's unusual and impressive here is the degree of logical and control of flow capabilities, as well as the blend of conventional technologies (like SSIS packages and stored procedures) and big data technologies (like Apache Hive Jobs, Python Scripts and Databricks notebooks).

A broad range of activities are on offer. Some of the more interesting ones are highlighted here

The pipelines are constructed visually, and even dragging a single activity onto the canvas allows a great deal of work to be done. In the figure at the beginning of this post, you can see that a single activity allows an entire Python script to be executed on Apache Spark, optionally creating the necessary HDInsight cluster on which the script will be run.

DataOps on board

ADF v2 also allows for precise monitoring of both pipelines and the individual activities that make them up, and offers accompanying alerts and metrics (which are managed and rendered elsewhere, in the Azure portal).

And as as good as the visual interface in v2 versus that of v1 is, ADF offers a range of developer interfaces to the service as well. These include Azure Resource Manager templates, a REST API interface, a Python option, and support for both PowerShellscripts and .NET code.

Behind the scenes

There are also elements to ADF v2 "under the covers" that are worth mentioning. For example, ADF v2 doesn't just support disparate data sources, but moves data between them at great scale when it uses its Azure Integration Runtime. (Microsoft says that ADF can move 1 TB of data from Amazon S3 to Azure SQL Data Warehouse in under 15 minutes.) Scale-out management of this facility is handled for the user by ADF itself. It's managed on a per-job basis and completely serverless, from the user's point of view.

ADF v2 also leverages the innate capabilities of the data stores to which it connects, pushing down to them as much of the heavy work as possible. In fact, the Azure Integration Runtime is less of a data transformation engine itself and more of a job orchestrator and data movement service. As you work with the individual activities, you find that much of the transformation work takes place in the data stores' native environments, using the programming languages and constructs they support.

Quo vadis

Compared to other data pipeline products, ADF v2 is both more evolved and less rich. Its innate capabilities are more limited, but its ability to connect, orchestrate, delegate and manage, using a combination of legacy and modern environments, is robust. It will provide Microsoft and Azure customers with an excellent data transformation and movement foundation for some time to come, and its accommodation of SSIS package lift-and-shift to the cloud makes it Enterprise-ready and -relevant right now.

How to Create Windows and Linux Virtual Machines (VM) on Microsoft Azure

Assuming you’ve already created a free trial Microsoft Azure account, you can now log in to create and prepare your Windows and Linux Virtual Machines for Acronis Backup 12.5.

Go to the Microsoft Azure Portal: https://portal.azure.com/ You’ll see the following Azure dashboard. Click the “+ New” at the top left corner.

Next, click “Compute.”

Let’s go with the latest technology! Select “Windows Server 2016 Datacenter.”

IMPORTANT! Make sure to select “Classic” from the “Select a deployment model” menu.

Enter the following details to identify your new VM:

Name: a unique name for your VM

User name: a VM administrator’s username that you’ll use to log in

Password: a new password

Subscription: Free Trial

Resource group: Select “Create New” and enter a descriptive name. If you put your future VMs in the same Resource Group, they’ll share the same local network, which will make things a lot easier.

Location: Choose an Azure Datacenter location closest to you.

IMPORTANT! Make sure that you remember the user name and password that you entered here! You’ll need them later to log in to your VMs.

Click “Next.”

On the next page, you’ll need to choose your VM resources. It’s safe to use the first available option, but please note that Trial Subscriptions have credit limitation and if you overuse the resources, your VMs will be disabled before the end of the free 30-day period.

Click “Select” to continue.

On the next page, just leave everything as is and click “OK.”

The next page will display a summary of all the settings that you entered. Click “OK.”

That’s it! You’ll be sent back to the main Azure Dashboard where you’ll see a blue square at the top right corner, showing the status of creating your VM. It will take a few minutes for the VM to be created.

Once provisioned, your VM will appear in the “All resources” widget. You only need to worry about the “Virtual machine (classic)” item. Don’t worry about the cloud service, storage account, virtual network etc. It just shows the resources your new VM is using:

Now, if you wish, repeat the steps to create additional VMs. For Windows, select “Windows 2016 Server.” For Linux, use “Ubuntu Server.”

PLEASE NOTE: some Linux operating systems don’t like long usernames. So, to be on a safe side and to prevent the VM creation process from failing, choose a short username (8 characters) when creating your Linux VM.

Step-by-Step Guide to Setting up a Windows Azure Free Trial

Microsoft Azure offers a free 30-day trial period to all new account holders.

Go to https://www.azure.com and click the green “Start free” button.

Next, click another “Start free” button.

If you already have an account with Microsoft, for example, Office 365, you’ll be prompted to log in.

When you log in, some of your details may already be there.

Follow the prompts to verify your account by phone (I used SMS).

You’ll also need to supply a valid credit card. Prepaid credit cards won’t work — you’ll need a “normal” credit or debit card. There is no charge involved with the setting up of a trial account. Microsoft just wants to see your card to verify your identity. There will be, however, a record for a $0 transaction on your bank statement. In my bank it looks like this:

Next – tick “I agree” and click “Sign Up.”

Within a few seconds, your account will be ready.

That’s it! Your Microsoft Azure account has been created.

To continue, click the “My Account” link at the top right corner or go straight to the Microsoft Azure Portal: https://portal.azure.com/