Custom C# Spark Jobs in Azure Synapse

Tutorial

Azure Synapse workspaces can host a Spark cluster. In addition to providing the execution environment for certain Synapse features such as Notebooks, you can also write custom code that runs as a job inside Synapse hosted Spark cluster.

This video walks through the process of running a C# custom Spark job in Azure Synapse. It shows how to create the Synapse workspace in the Azure portal, how to add a Spark pool, and how to configure a suitable storage account. It also shows how to write the custom job in C#, using Visual Studio Code's support for Docker-based development to target the exact JDK and Scala version used in Synapse. You will see how to upload the built output to Azure, and then how to configure Azure Synapse to execute the resulting JAR as a custom job.