Engineering Practices

This post gives an overview of Continuous Integrations and shows how you can implement it with GitHub Actions, with an accompanying example Python project

Publishing individual scripts to the PowerShell Gallery can be an effective way to share pervasive functionality without the overhead of maintaining a full-blown PowerShell module. In this post we'll look at how we can take a function that is already published as part of a module and also make it available as a standalone script, without having to maintain two versions of the code.

Have the recent changes to Azure CLI on Windows broken your dev inner loop when working with containerised applications that need access to Azure resources? If so, this post provides a workaround that will unblock you without having to be stuck on an old version of the Azure CLI.

In this blog we demonstrate how the Gherkin specification can be adapted to enable BDD to be applied to data engineering use cases.

We often come across spreadsheets and CSV files that contain features which are a list of tags or categories. This blog article walks through a simple way of extracting insights from this data using a combination of Pandas for data wrangling and Power BI for analytics and visualisation.

Despite being inherently difficult to test, the need to validate data modelling, business rules and security boundaries in Power BI reports is important, as well as the need for ensuring that quality doesn't regress over time as the insights evolve. This post explains that, by executing DAX queries over underlying tabular model, it is possible to execute scenario-based specifications to add quality gates and build confidence in Power BI reports, just as any other software project.

Containers and Docker offer a different approach to development and deployment of applications. This post provides an introduction to containerisation and Docker, and provides examples of creating containerised applications with Docker.

The new Power BI ExecuteQueries REST API presents a number of new opportunities for Power BI developers in terms of tooling, process and integrations. This post highlights some of the key advantages of this new capability.

This blog post explains how to set up multiple instances of a Linux distribution in the Windows Subsystem for Linux.

Bicep provides a significant improvement in tooling for organisations seeking to deploy infrastructure as code on Microsoft Azure.

Many companies struggle to bridge a communication gap between their business stakeholders, and the technical team. At endjin we employ a few different techniques in order to help our customers overcome this challenge. These include (but aren't limited to) user stories, BDD specs, and a ubiquitous language. This blog focuses on user stories, and how these can be used to enable business-wide communication or requirements.

Interactive Spark notebooks are an incredibly powerful tool for data exploration and experimentation. And in Azure Synapse, the time to (business) value is significantly decreased due to tight integration with Pipelines and monitoring tooling. But as with any software process, the need to validate business rules is important, as is ensuring that quality doesn't regress over time - especially so in such a collaborative and productive environment. This post looks at some simple ways to add testing to your Synapse Notebooks.

Data insights are useless, and even dangerous (as we've seen recently at Public Health England) if they can't be trusted. So, the need to validate business rules and security boundaries within a data solution is critical. This post argues that if you're doing anything serious with data, then you should be taking this seriously.

This post describes the reasons for choosing specific & immutable dependency versions in your software.

GitOps (sometimes known as 'operations by pull request') is most often discussed as being an approach for managing your Kubernetes platform, but in this post I will make the case for how it can benefit a much wider audience.

Often when building software we need to outsource part of the solution to a third party. Common examples include authentication, payment, and email, but there are many others. When first designing a solution the decision around which provider to use can be difficult to tackle. At endjin we have developed a system for answering the question "which provider should I use?".

Many API frameworks implement an async HTTP API pattern to address the problem of coordinating the state of long-running operations with external clients. This post looks at how we can use Axios interceptors to simplify and centralise the UI logic required to work with this API pattern.

Azure AD B2C is easy to integrate and configure for simple sign-in / sign-up authentication. But there's a step change in complexity as soon as you want to do something outside of the built-in user flows. An "invite user" flow is one such example, but it's also a fairly common requirement in any business or team orientated SaaS application, which makes AAD B2C as less attractive choice. This post describes an alternative approach for this authentication feature that avoids the need for AAD B2C custom policies, allowing you to keep authentication as simple as possible.

Ian Griffiths is talking about the Reactive Extensions for .NET at dotnetsheff. Learn the ideas behind Rx, and see it applied in an IoT applicable to upload clinical monitoring data to the cloud.

This blog post explains how you can automatically open a web browser in either private browsing mode or with a custom user profile when debugging web applications from Visual Studio.

Whilst "read/write XMLA endpoint" might seem like a technical mouthful, its addition to Power BI is a significant milestone in the strategy of bringing Power BI and Analysis Services closer together. As well as closing the gap between IT-managed workloads and self-service BI, it presents a number of new opportunities for Power BI developers in terms of tooling, process and integrations. This post highlights some of the key advantages of this new capability and what they mean for the Power BI developer.

C# 8.0's nullable references feature dramatically changes a fundamental aspect of the language. In this post, Ian explains how to you can soften the impact by enabling gradually across your projects.

Despite being inherently difficult to test, the need to validate data modelling, business rules and security boundaries in Power BI reports is important, as well as the need for ensuring that quality doesn't regress over time as the insights evolve. This post explains that, by connecting to the underlying tabular model, it is possible to execute scenario-based specifications to add quality gates and build confidence in Power BI reports, just as any other software project.

Whilst testing Power BI Dataflows isn't something that many people think about, it's critical that business rules and associated data preparation steps are validated to ensure the right insights are available to the right people across the organisation. Data insights are useless, even dangerous, if they can't be trusted, so despite the lack of "official support" or recommended approaches from Microsoft, endjin treat Power BI solutions just as any other software project with respect to testing - building automated quality gates into the end to end development process. This post outlines an approach that endjin has used to test Power BI Dataflows to add quality gates and build confidence in large and complex Power BI solutions.

Azure Analysis Services provides an enterprise-grade analytical platform with massive scale and flexibility. But, as one of the more expensive services in the Azure platform, consideration should be given to cost management, especially in multi-environment ALM scenarios. This post explains how to massively reduce running costs through automation using Powershell and orchestration tools like Azure DevOps.

At endjin, our approach focuses on using scientific experimental method to support the creation of fully proved and tested decision making, and the use of scientific research to support our work. This post runs through how we applied that process to creation a pipeline to detect vessel proximity.This is an example which is based around the project we recently worked on with OceanMind. In this project we helped them to build a #serverless architecture which could detect vessel proximity in close to real time. The vessel proximity events we detected were then fed into machine learning algorithms in order to detect illegal fishing!Carmel also runs through some of the actual calculations we used to detect proximity, how we used #data projections to efficiently process large quantitities of incoming data, and the use of #durablefunctions to orchestrate the processing.

During our projects at endjin, we often find ourselves evangelising Behaviour Driven Development, and specifically SpecFlow. In this post we look at a technique for defining complex test data objects in your Gherkin feature files, which we've also made available via the endjin-sponsored Corvus.NET project.

This post explains how to update Azure Analysis Services model schemas from inside custom .NET applications. Whilst not a common scenario for most, it shows that this is easy to do using the AMO SDK. So, there's nothing stopping you from developing complex and rich end-user functionality over the top of your data analysis solutions - providing run-time, user-driven schema changes like "what if" analysis.

See how to manage consistent default configuration across all your .NET projects by using NuGet build assets.

If you use Azure Functions on a regular basis, you'll likely have grappled with the challenge of testing them. Even now, several years after their introduction, the testing story for Functions is not hugely well defined. In the final post in this series, we show how to ensure specs written using Corvus.Testing can run as part of your build pipeline.

If you use Azure Functions on a regular basis, you'll likely have grappled with the challenge of testing them. Even now, several years after their introduction, the testing story for Functions is not hugely well defined. In the fourth of this series of posts, we look at how configuration can be supplied from your tests to the functions apps being tested.

If you use Azure Functions on a regular basis, you'll likely have grappled with the challenge of testing them. Even now, several years after their introduction, the testing story for Functions is not hugely well defined. In the third of a series of posts, we look at using classes in the Corvus.Testing library to run functions apps via scenario and feature hooks.

If you use Azure Functions on a regular basis, you'll likely have grappled with the challenge of testing them. Even now, several years after their introduction, the testing story for Functions is not hugely well defined. In the second of a series of posts, we look at using step bindings provided by the Corvus.Testing library to run functions apps as part of your SpecFlow scenarios.

If you use Azure Functions on a regular basis, you'll likely have grappled with the challenge of testing them. Even now, several years after their introduction, the testing story for Functions is not hugely well defined. In the first of a series of posts, we look at some different approaches to testing your functions apps, and introduce the Corvus.Testing library.

In our recent project with OceanMind we used #AzureFunctions to process marine vessel telemetry from around the world. This involved processing huge quantities of data in close to real time. We optimised our processing for a serverless environment, the outcome of which being that the compute would cost less than £10 / month!This post summarises some of the techniques we used, including some concrete examples of optimisations we made.

C# 7.2 introduce ref structs, a new kind of type (Span<T> is a ref struct) designed to support certain high performance scenarios. There are constraints around their use, and when writing unit tests for our Ais.Net parser, this caused some challenges. This blog describes the technique we used to work around the constraints.

The application of scientific experimental process to software development leads to the development of fully-validated solutions. This approach provides you with confidence in designs and means that you can quickly identify ideas which are not worth pursuing. At endjin we use the ideas of hypotheses and experimentation when designing any solution and this gives us full confidence in the designs we produce. In this post we outline the steps and advantages of using this approach.

Integrating Azure Analysis Services into custom applications doesn't just mean read-only data querying. But if your application changes the underlying model, it will need to be re-processed before the changes take effect. This post describes how to use the REST API for Azure Analysis Services inside a custom .NET application to perform asynchronous model refreshes, meaning your applications can reliably and efficiently deal with model updates.

If you've worked with Power BI at all, you'll have probably realised that there are numerous mediums through which you work with (potentially the "same") data. Data types across these mediums can be called different things, but actually refer to the same thing. They can also (unsurprisingly) be called different things and actually mean different things. It's useful to know what the corresponding data types are across these mediums, as you may need to, for example, convert queries from one format to another. This blog and containing report intend to clarify what the corresponding data types are across each of the separate mediums within Power BI.

There are hidden pitfalls with dependency injection, particularly when managing the lifetime of scoped components. What is safe? And are there other approaches we can take to managing scoped object lifetimes?

Being able to construct DAX queries dynamically in C# means the possibilities are endless in terms of integrating Azure Analysis Services queries into your custom applications, and with the code samples in this post, you have everything you need to get started.

The Azure CNAB Quickstarts Library helps you get up and running with CNAB and Porter. We've built quickstarts covering solutions like WordPress, Ghost, Mattermost, and data platforms like Apache Airflow, SQL Server AlwaysOn clusters and Kubernetes features like an nginx ingress controller and an Azure AD enabled OAuth2 Proxy. We've condensed all our learnings from the past 9 months of working on the project and turned them into a 10 minute video which explains all the key concepts. We hope this video helps accelerate your own CNAB & Porter epiphanies!

In partnership with Microsoft, we have released Azure CNAB Quickstarts Library on GitHub. CNAB (Cloud-Native Application Bundle) is a new specification designed for facilitating the packaging, installation, upgrading and uninstallation of cloud-native solutions in the cloud, on-premise or on the edge. We've created a number of quickstarts covering Apache Airflow, Azure Kubernetes Service, Ghost, Kubeflow, SQL Server Always On and Wordpress to help demonstrate the power of CNAB and Porter.

Integrating Azure Analysis Services into custom applications means more than just querying the data. By surfacing the metadata in your models, you can build dynamic and customisable UIs and APIs, tailored to the needs of the client application. This post explains how easy it is to query model metadata from .NET, so you can create deeper integrations between your data insights and your custom applications.

One of the first steps in integrating Azure Analysis Services into your applications is creating and opening a connection to the server - just like any other database technology. This post explains the ins and outs of creating Azure Analysis Services connections, including code samples for each of the key scenarios.

In this post, Ian describes some of the highlights from the NDC London conference

GitHub Actions is GitHub's new CI/CD platform. It is comparable with Azure Pipelines, which forms part of the Azure DevOps suite. In this post, Mike Larah looks at the similarities and differences in the high-level concepts and terminology between the two platforms.

This is the second blog in a series around design patterns. This post focuses on the builder pattern. The builder pattern is used when there is complex set up involved in creating an object. Like the other creational patterns, it also separates out the construction of an object from the object's use.

Whilst some of the Azure Active Directory PowerShell for Graph module (AzureAD) functionality has been rolled into the new Azure PowerShell Az module, it's not currently (and might never be) a replacement for the full power of what you can achieve with AzureAD. So, there's every chance you'll find yourself needing to use both side-by-side. This post explains how to do that using the new cross-platform PowerShell Core.

A Power BI based solution typically consists of a variety of technologies - for example Azure data platform services containing source data. As such, automation of Power BI resources needs to be considered as part of a wider DevOps strategy. This post describes the specific steps needed in order to fully automate the creation and security of Power BI workspaces using Powershell and Azure DevOps pipelines.

Software architecture and Agile project planning are often seen to be at odds. However, here at endjin we think that the way in which they intersect solves a lot of the common issues surrounding architecture. The key to successful architecture is constantly keeping the drivers in mind and having a tight communication loop as the architecture is implemented. These concepts are ones which are key to agile project management, and the combination of these two disciplines can be extremely powerful and successful when applied correctly.

Creating a architecture in an iterative and incremental solution delivery is challenging. I've evolved an approach akin to Painting By Numbers, and am inspired by a quote from the artist and film maker Dave McKean.

At endjin we have a high quality bar when it comes to our code. As part of this we carry out regular code reviews. One of the tools we have used for these code reviews is NDepend. This is the second in a blog series written as we carried out that process. This post focuses on the insight you can quickly gain just by glancing at the NDepend UI.

This blog post details how we used Azure Policy to enforce Azure resources were tagged with appropiate tags and ensured tags were inherited from parent resource groups where possible.

At endjin we have a high quality bar when it comes to our code. As part of this we carry out regular code reviews. One of the tools we have used for these code reviews is NDepend. This is the first in a blog series written as we carried out that process. This post runs through the different metrics used by NDepend, and the reasons that each of these can be an indication of code quality.

There is a lot of hype around AI & ML. Here's an example of using ML.NET & Azure Functions to deliver a series of micro-optimisations, to automate a series of 1 second tasks. When applied to business processes, this is what the 4th Industrial Revolution could look like.

When he joined endjin, Technical Fellow Ian sat down with founder Howard for a Q&A session. This was originally published on LinkedIn in 5 parts, but is republished here, in full. Ian talks about his path into computing, some highlights of his career, the evolution of the .NET ecosystem, AI, and the software engineering life.

This is the second blog in a series which delves into how the Rx operators work under the covers. This series aims to provide a greater understanding of Rx and its operators. This post focuses on the AGGREGATE operator.

Two of the main use-cases for explicit interface implementation are: 1. to hide members of the interface in a class which inherits from that interface, and 2. to work around the scenario when a class is inheriting from two interfaces which share a member of the same name. Take a look at this blog to go into more depth about each of those scenarios.

A good test suite should validate behaviour across your code base, testing as many edge cases as reasonably possible. A common "edge case", is passing a null value when a value isn't expected to be null. In Specflow, all values used in scenario examples are treated as strings, so it isn't possible (by default) to pass in a null value. However, using "Step Argument Transformations", we can achieve the desired behaviour. Read this blog to learn how this can be implemented.

At endjin we've done quite a lot of work around reactive data processing. This post is a great introduction to the main concepts in reactive programming. It runs through the main classes in Rx.NET and provides an easy-to-follow explanation of this powerful (and somewhat complex) area of .NET.

This is the second blog in a series about data flow. This post delves into TPL dataflow.The task parallel library is a .NET library which aims to make parallel processing and concurrency simpler to work with. The TPL dataflow library is specifically aimed at making parallel data processing more understandable via a pipeline-based model.

Combining deferred execution with asynchronous operations can cause subtle problems. This post shows how to ensure correct operation when these idioms combine.

This is a post about the importance of reducing your dependencies as much as possible when designing solutions.If as much as possible is under your own control, there is far less room for synchronization errors and update conflicts.

Ian Griffiths recently joined endjin as a Technical Fellow. We had a long fireside chat, which has been broken down into a 5 part series covering .NET, the Cloud, AI, ML, teaching software engineering, and why he joined endjin.

Everyone learns differently. In this post Carmel describes how OneNote can be used to aid and enhance research. As an avid note taker and blogger, she highlights how the ability to Ctrl-F into written text in OneNote has greatly improved her productivity!

Garbage collection is the process in .NET of cleaning up unused memory. Here is a great post which explains how it works under the covers. If you have ever wanted to get an understanding of exactly what is going on behind the scenes when you write C# code, then this is the post for you!

As an apprentice engineer at endjin, you cover a lot of ground, especially at a consultancy which works with the latest and greatest tools to solve its clients' problems. Learn about endjin's Modern Data Platform, which is a culmination of IP, processes and knowledge built from years of implementing high-performance data-driven solutions. Also learn about the types of tools an apprentice gets to use, and the types of things an apprentice learns along the way.

Using the recently Open Sourced OpenAPI SDK, we create a Visual Studio extension from converting OpenAPI specificaitons between YAML and JSON formats.

The evolution of .NET is a complex one. We've put together a deck to help you understand what you should choose to use when.

One very useful but little used pattern when working with Resource Manager templates, is the ability to use parameters to optionally deploy resources, constrain certain resource configurations based on other user defined parameters, or to toggle parameters based on other values. This blog post explores some of these options.

Did you know that you can fully automate the bootstrapping process of setting up an Azure DevOps enviornment? This post shows you how.

In the world of DevOps, cloud and platform services, how does a developer's "definition of done" need to change? This post argues that as the silos of development and operations are broken down, the responsibility of understanding the whole solution increases meaning, to truly take advantage of the cloud, the need for quality and professionalism is critical for success.

Learn how to combine the power of Azure Resource Manager and Octopus Deploy for a frictionless Azure DevOps experience.

Azure Machine Learning Studio is a fantastic service for experimentation, but you can also easily productionize your machine learning models. In this post we show how to create an Azure ML Studio web service and test it using Postman.

Meta-Runners allow you to easily create reusable build components for TeamCity, in this post I demonstrate how to create a Meta-Runner to create Azure Application Insights Release Annotations.

Alice reflects on year 2, being given more responsibility, diving deeper into all aspects of software delivery, and the good habits she's been building.

What do machine learning and data science actually mean? This post digs into the detail behind the endjin approach to structured experimentation, arguing that the "science" is really all about following the process, allowing you to iterate to insights quickly when there are no guarantees of success.

Mike Larah shares his tips for how to best customize ConEmu to improve your terminal experience for Windows Command Prompy and PowerShell

We recently upgraded our TeamCity based CI/CD environment, but we had to make some adjustments to fix failing tests.

Meta-Runners allow you to easily create reusable build components for TeamCity, in this post I demonstrate how to create a Meta-Runner to wrap ILMerge functionality.

Learn how to authenticate against Azure Resource Manager using PowerShell, for fully automated deployments.

We've built and open-sourced a Visual Studio Extension that provides basic support for discovering and running CucumberJS tests with Node.JS, in the Visual Studio Test Environment.

Running a start-up or bootstrapping a new product within an established business can feel pretty chaotic. We talk about strategies for managing the time of a team, and ensuring that you become predictable.

This is a quick guide to setting up cucumber JS in Visual Studio with grunt.

Git has powerful extensibility points, which you can harness to automate your dev inner loop. In this post we examine how to use Git Hooks to reduce the occurrence of merge conflicts.

We use SpecFlow to write integration tests against Azure. In this post we share an Open Source SpecFlow extension for the Azure Storage Emulator.

One of the great benefits of using SpecFlow is that it allows you to write your specifications in a human readable format. Learn how you can create reusable step argument transformations to apply custom transformations to your parameters, so that you can keep your specifications easy to read.

Hiring developers is hard. Really hard. Here are some tips on how to make it easier on yourself.

To help guide the client through the selection process (and to act as an aide memoire when we come to try and remember why particular choices were made), we created a handy flow-chart to take you through the technology landscape, and suggest some recommended-practice solutions.

We've delivered a number of Elasticsearch solutions on Azure. In this second post in the series, we share some useful tips.

In this post, we explore the use of SideWaffle to create a frictionless experience for developers wanting to create repeatable plugins.

There's a lot of documentation available around NuGet and how to create/publish packages, etc. But when I looked for a simple step-by-step guide on how to test a package locally, I couldn't find any.

You can easily notify NewRelic that a deployment has occured. In this post I demonstrate how to turn a script into a reusable TeamCity Metarunner.

Semantic logging enables much richer diagnostics and analysis. It is supported in Windows and in .NET. This post explores the problem space. Others in the series explore how we can use Reactive Extensions to enable some powerful scenarios.

Delving a little deeper into Behaviour Driven Development and Specification by Example.

In this blog post, I provide a how-to guide for automating your TeamCity backups using a PowerShell script.

Sometime upgrading in-place is no longer an option and you need to migrate to a new server. This step-by-step guide takes you through the process of migrating a TeamCity instance.

Learn why we created the endjin composition framework for dependency injection.

In the final part of this series, we cover how to use TeamCity's feature branch abilities to support the GitFlow process.

This post finally covers the nitty-gritty of how you use the GitFlow extensions to drive the workflow.

In this post we show you how to get up and running with GitFlow.

Adopting Git means creating an entirely new mental model about how to version control your source code. Once you make the jump, there's no going back.

A quick example of how to set up integration testing for HttpClient and HttpServer

Mike starts his first day as an Apprentice with an introduction to PowerShell and being onboarded to endjin's various tools and platforms, while attempting to automate the entire process.

To celebrate joining JetBrains Development Academy Board, we release a PowerShell modules to make working with TeamCity easier.

PowerShell is fantastic language for DevOps, automation and general scripting. Every developer should be familiar with it; here's a guide to some useful resources to help you learn it.

Long time listener, first time guest. I was invited to speak on the .NET Rocks! Podcast, to talk about Sharp Architecture and various other Open Source projects I've been involved with.

We've decided to combine the StyleCop and StyleCop for ReSharper code bases to make long term maintenance easier.

In this step-by-step how-to guide, we take you through the process of setting up YouTrack, and using IIS as a reverse proxy so you can access your YouTrack server via a custom domain.

In this step-by-step how-to guide, we take you through the process of setting up TeamCity, and using IIS as a reverse proxy so you can access your TeamCity server via a custom domain.