Year 2 as a software engineering apprentice at endjin
In February 2016, I completed my second year of endjin's three year custom apprenticeship scheme. This blog is a chance for me to reflect on what was learnt over the year - hopefully others will find it useful too.
Year one had involved a very steep learning curve as I transitioned from student to graduate to real world apprentice, and learned about C#, Azure, and our working practices and tools.
At the end of the year I wrote a retrospective and set out some aims for year two; to deepen my technical knowledge, learn about best practices for architecting solutions and the intersection between architecture and the choice of Azure services, and to carry on exploring Azure's data processing technologies.
I hadn't consciously kept all these aims in mind throughout the year, but they are fairly close to what happened, although there is still a huge amount to learn in all these areas. The main themes of year two were increased responsibility for the delivery of projects, lots of automation, data services, a company-wide focus on working processes, and career reflection.
More responsibility for delivering projects
This is probably the area you'd expect to hear about in a year two apprenticeship post.
In year one I'd worked closely with more senior developers to develop Azure Web Apps and Cloud Services.
In year two, I worked with greater independence on a number of projects, including two fairly large pieces of client work - upgrading a video recording mobile app, and adding favouriting functionality to a web application, covering the full project lifecycle from user story elucidation, to estimation, to sprint planning, going through the sprints, and finally release.
Working on the mobile app, which was developed using Xamarin across the three major platforms, was particularly interesting because this required some significant set up and a different approach to debugging, testing and release.
For a while my desk was occupied by a bank of test devices and a borrowed MacBook. Overall, Xamarin made developing an app across iOS, Android and Windows Phone easier, but because of the complexity of the problem (attempting to keep the core functionality of a non-trivial video recording app in a shared project used by all platforms), there were a few difficult moments, especially around package upgrades.
The release process was another hurdle, in particular the extra steps required to emulate deploying to a test environment for each platform. It was my first exposure to the MVVM pattern.
The mobile app was supported by a Cloud Service and Azure Media Service, causing me to get better acquainted with Cloud Service interactions with Azure Storage Queues and media encoding, and best practices for cloud architecture.
During the project, at certain points I hit barriers regarding the encoding service and contacted the Media Services team to raise bugs and queries, which were fixed and answered.
The lesson taken from this (and deep in the company culture) is to take a pro-active, systematic approach to dealing with serious 'blockers'.
If something appears to be wrong with a service beyond our control, after a reasonable amount of research to establish what is happening, we would raise the issue to the owner of the service, providing them with sufficient details and background to investigate it (more on this later).
The project to build favouriting functionality for logged in and anonymous users of a web application was more familiar ground. It was a satisfying project to work on as it brought together front and back end technologies and helped to consolidate some AngularJS, C#, MVC and Elasticsearch concepts, as well as our branching, code review and web deployment process.
Alongside these longer projects I also worked on technical spikes, new features and bug fixes for Web Apps, Cloud Services, the Mobile App and endjin's own libraries.
Another aspect of having more responsibility for delivering projects was that I was now able to provide client developer support and training on topics like GitFlow, Team City, Media Services, Cloud Services and Web Apps, which I enjoyed.
Automation
A repeated theme during year two was automation. In some cases, projects were explicitly aimed at automating manual processes, such a piece of work to use Azure Automation to start and shut down VMs at certain times.
In other cases, automation was put into place to speed up our work, and reduce repetition. For example, I built a tool which took and compared screenshots of a web app at the main bootstrap screen sizes and output a heat map of areas which were different, to help our design team with a CSS to LESS migration, set up automated testing using SauceLabs, Selenium & Team City, and helped to develop a templating service for CMS plugins.
Azure Data Services
Early in year two, I developed an application to automate the generation of endjin's Azure Weekly newsletter, using Azure Machine Learning, Stanford NLP libraries, and endjin text scoring library. Later in the year I worked with a colleague to re-factor this application into a generic news aggregation and publication platform.
We used Data Factory and Custom Activities implemented using TPL Dataflow to replace/ augment the now-defunct Yahoo Pipes, fetching in and enriching articles before they are exposed for editing and publication via a Web App.
I also worked with colleagues to endjin's Online Counterfeit Goods Detection Service and re-factor the service's core matching functionality for use on Azure Batch, and created PowerBI dashboards to display metrics based on Azure Table Storage data.
Towards the end of year two, following some intensive R training from Revolution Analytics (now part of Microsoft), covering R Open, R Enterprise and Deploy R, I've been working on an increasing number of data projects involving Azure Machine Learning experiments and web services, HDInsight, Azure Batch, R, and Data Factory.
Azure
Continuing to work on Azure-related projects helped to deepen my knowledge this year. In addition to work-based learning, a group of us were given time to prepare for the Developing Microsoft Azure Solutions (70-532) exam, which helped to formalise some of the knowledge I'd picked up over the year and fill in some gaps.
It covers Web Apps, VMs, Cloud Services, Storage (Blobs, Tables, Queues, SAS, and Azure SQL), Active Directory, Networking, Service Bus and Redis Cache, and confirmed that I knew a lot more about Web Apps, Cloud Services and Storage than I did about VMs, Service Bus or Networking (and network configuration for VMs, Cloud Services etc).
The exam was pretty good at selecting for practical experience of all the services (as opposed to just reading the book and doing quiz type revision). Wish me luck for the re-take this year…
Azure services change very frequently, so much that none of the topics mentioned above will remain the same for the full year. For example, Azure Resource Manager is changing the landscape, and deprecating some of the roles performed by Cloud Services. Helping to produce our newsletter is a good way of keeping up to date with new developments.
Process
Moving away from the purely technical, a lot of what I learnt this year was related to working processes. Process is really important at endjin, and is reflected in our project review system and bonus scheme. Because I worked with greater independence during year two, I had to take more responsibility for this aspect of work. Here's a quick summary of some of them:
Pair where possible
Pairing with a colleague brings huge benefits in terms of framing the aims of a session well, focus, pace of work, and time spent on problem solving. It also makes the job more sociable and rewarding. When working on machine learning projects, there's an extra benefit that one person can be writing up lab notes on a laptop while the other drives.
Track all work
All work is tracked with a work item. Endjin is very transparent about the financial aspects of each project, so we understand the importance of tracking work to monitor the remaining time available and accurately carry out financial reporting. The year included an Accounting 101 session from Alan our accountant, which joined up some dots about how this data is used.
Send a video update to clients at the end of each week
Client communication is as important as the work that you do for the client. We send out a short video progress report for each week's work, to keep clients informed and get their feedback. This has proved very popular. It also helps to keep us up to date with what colleagues are working on, and provides a reference when we are looking back at past projects.
Escalate all issues with third party service providers
All issues or questions regarding Azure or other third party services, which can't be solved with reasonable effort, should be raised with the service provider. In the case of Azure services, we raise them with the relevant team using the Azure Advisers forum, so we can track these issues and help the product teams.
It seems like a lot of work at first because you have to spend time collecting data, screenshots etc. and checking your facts when you just want to get on with working on the project at hand, but in fact this acts as a good validation process for your hypothesis about the situation.
I sometimes end up finding the answer to a problem while writing up the issue for Azure Advisers, or realise the issue is more nuanced than I'd first thought. Another barrier I had initially to raising issues was thinking that an issue was too insignificant to bother people with - after all there are open forums full of posts.
I've been trained this year to ignore any such qualms, as long as it's reported with sufficient information and background for the service provider to do something with it. On a similar note, I try to vote on requested features every time a search for a particular piece of functionality we need ends up at a poll.
Harvest re-usable work
Pieces of work that can be re-used are tracked and harvested to save future effort. Where possible, they are identified when they are created, then harvested during any periods of downtime.
Work sustainably
We are encouraged to keep to sustainable working hours and patterns. During year one I'd slipped into often working late (doing my MSc at the same time didn't help), which is fairly common in the industry, but frowned upon here.
One major achievement this year is to get my working hours under control, and have a lot more free time and clarity as a result.
I was helped in this by attending Springboard, a women's development course. The course consisted of a series of workshops attended by a small group of women who worked in various sectors, and associated homework.
We were encouraged to reflect on our values, and identify how they were met in different aspects of our lives, which was a really useful exercise I'd encourage anyone to do. We also met some very determined and interesting speakers who gave us insights into how they'd got to where they were professionally and personally.
Recruitment from another perspective
After being on one side of the recruitment process quite recently, I got to see it from the other side this year, as we carried out a large recruitment drive.
Our recruitment process involves all office-based staff, and consists of an initial CV and letter screening, followed by the person who applied making a day long visit to the office, pairing with as many of us as possible, and going out to lunch.
We were all involved in writing up feedback to the applicant which was passed back to them each time.
It made me realise how important factors like self-awareness, a good temper and a genuine like of other people are to employers, assuming a certain level of technical ability.
Another insight from the process was that technical ability must be matched by an interest in the wider aims driving any piece of work – at least for working here.
I volunteered to organise some of our graduate recruitment by making contact with a number of university careers services, of which, the Birkbeck Careers Service was outstandingly helpful and provided some good candidates.
The future
At the end of year one, I'd been feeling rather fearful about the power of the role of developer and its potential to cause harm.
No matter how many tests we had in place and how smooth the actual process, deployments were a stressful experience and I'd worry long afterwards about every possible thing I could think of that might go wrong. The fear hasn't gone away (and probably shouldn't), but it's got a bit better with experience from this year.
I think this is because the better you understand a system, the more you can move away from general anxiety about unknowns, and focus instead on dealing with specific areas of risk.
Over year two, I've particularly enjoyed working on short spikes to join together services in a new and interesting way, working on data services that take raw input and produce beautiful graphs, communicating about technology through reports, blogs and training, and pairing with others on just about anything.
In year three, I'm hoping to do more of all these things, and further develop my data science, Azure, architectural and .NET knowledge.
If you are interested in finding out more about endjin's apprentice scheme, you can go to our recruitment page, or get in touch at apprentices@endjin.com
