How to trigger an Azure Synapse pipeline run from C#
As part of a reporting project we've been working on recently, we needed to be able to kick off an Azure Synapse pipeline run from C# code running in an Azure Functions app. Documentation is a bit sparse in this area, so here's how to do it.
How do I do it?
- Add a reference to Azure.Analytics.Synapse.Artifacts. You can find the API documentation here. This provides the C# API for talking to Synapse. At the time of writing, I used the
1.0.0-preview.4version. - In your code, create a new instance of the Azure.Analytics.Synapse.Artifacts.PipelineClient class. This requires the dev endpoint for your Synapse instance, as well as your preferred means of authentication. More on this below.
- Assuming your pipeline needs some parameters supplying, create a
Dictionary<string, object>containing them. - Execute PipelineClient.CreatePipelineRunAsync() method to kick off the pipeline run.
Show me the code!
var pipelineClient = new PipelineClient(
new Uri("<your dev endpoint here>"),
new DefaultAzureCredential());
var parameters = new Dictionary<string, object>
{
{ "parameter1", "value1" },
{ "parameter2", "value2" },
};
Response<CreateRunResponse> run = await pipelineClient.CreatePipelineRunAsync(
"<your pipeline name here>",
parameters: parameters).ConfigureAwait(false);
// The response contains the Id of the new pipeline run.
Console.WriteLine(run.Value.RunId);
The dev endpoint you need when creating the PipelineClient can be found in the portal, and has the form
https://<your synapse instance name>.dev.azuresynapse.net
How does my code authenticate to the Synapse API?
Authentication is via the Azure.Identity library, so you have the standard options. DefaultAzureCredential is probably the one you'll need - it works through a list of authentication mechanisms in order of likelihood (you can see the list here), starting with credentials stored in environment variables and ending with an interactive browser login prompt.
If like me your code is running in an Azure Functions app with a Managed Identity (and assuming you don't override it using environment variables), this means that the Managed Identity of the Functions app will be used to authenticate to the Synapse endpoint. And when running locally, it should pick up your Visual Studio or Azure CLI credentials and use those.
Either way, in order to run the pipeline, that identity will need to have been granted the Synapse Credential User role in your Synapse instance. This can be done using the "Access Control" panel in the Management area of Synapse Studio (note the warning when you grant access - it can take a few minutes for the update to take effect).
