Skip to content
Jonathan George By Jonathan George Software Engineer IV
How to trigger an Azure Synapse pipeline run from C#

As part of a reporting project we've been working on recently, we needed to be able to kick off an Azure Synapse pipeline run from C# code running in an Azure Functions app. Documentation is a bit sparse in this area, so here's how to do it.

How do I do it?

  1. Add a reference to Azure.Analytics.Synapse.Artifacts. You can find the API documentation here. This provides the C# API for talking to Synapse. At the time of writing, I used the 1.0.0-preview.4 version.
  2. In your code, create a new instance of the Azure.Analytics.Synapse.Artifacts.PipelineClient class. This requires the dev endpoint for your Synapse instance, as well as your preferred means of authentication. More on this below.
  3. Assuming your pipeline needs some parameters supplying, create a Dictionary<string, object> containing them.
  4. Execute PipelineClient.CreatePipelineRunAsync() method to kick off the pipeline run.

Show me the code!

    var pipelineClient = new PipelineClient(
        new Uri("<your dev endpoint here>"),
        new DefaultAzureCredential());

    var parameters = new Dictionary<string, object>
    {
        { "parameter1", "value1" },
        { "parameter2", "value2" },
    };

    Response<CreateRunResponse> run = await pipelineClient.CreatePipelineRunAsync(
        "<your pipeline name here>",
        parameters: parameters).ConfigureAwait(false);

    // The response contains the Id of the new pipeline run.
    Console.WriteLine(run.Value.RunId);

The dev endpoint you need when creating the PipelineClient can be found in the portal, and has the form

https://<your synapse instance name>.dev.azuresynapse.net

Screenshot of the Azure Portal with the necessary endpoint information highlighted

How does my code authenticate to the Synapse API?

Authentication is via the Azure.Identity library, so you have the standard options. DefaultAzureCredential is probably the one you'll need - it works through a list of authentication mechanisms in order of likelihood (you can see the list here), starting with credentials stored in environment variables and ending with an interactive browser login prompt.

If like me your code is running in an Azure Functions app with a Managed Identity (and assuming you don't override it using environment variables), this means that the Managed Identity of the Functions app will be used to authenticate to the Synapse endpoint. And when running locally, it should pick up your Visual Studio or Azure CLI credentials and use those.

Either way, in order to run the pipeline, that identity will need to have been granted the Synapse Credential User role in your Synapse instance. This can be done using the "Access Control" panel in the Management area of Synapse Studio (note the warning when you grant access - it can take a few minutes for the update to take effect).

The best hour you can spend to refine your own data strategy and leverage the latest capabilities on Azure to accelerate your road map.

Jonathan George

Software Engineer IV

Jonathan George

Jon is an experienced project lead and architect who has spent nearly 20 years delivering industry-leading solutions for clients across multiple industries including oil and gas, retail, financial services and healthcare. At endjin, he helps clients take advantage of the huge opportunities presented by cloud technologies to better understand and grow their businesses.