Updating dependencies in Azure DevOps repos
Background
One problem we have as developers, is keeping on top of our dependency housekeeping. How often have you seen a repo with packages that are months, perhaps years, out of date? And how many times have you tried to update one or more, only to rage-revert after hours of hair pulling frustration? Updating these dependencies few and often is most definitely the way forward.
In my last post I wrote about Creating a Pull Request workflow in Azure DevOps; I’ve since applied this to many of the repos belonging to the client I’m currently working for. I want to be a good citizen, leaving these in good shape for the client’s developers when my contract ends and I move on; semi-automated dependency housekeeping was on my TODO list.
Dependabot is a tool for such housekeeping, and having been recently acquired by GitHub it has become free of charge. Dependabot publish Ruby Gems from dependabot-core that you can run in your own environment — which I will demonstrate here — as an alternative to their own hosted experience.
I decided to write this post as Azure DevOps support in Dependabot is relatively new, and as such online resources are few and far between. Dependabot provide the dependabot-script repo which is the only resource I could find and has guided me most of the way.
TL;DR
If you’d rather just dive in, simply import my azure-dependabot repo into your Azure DevOps project and follow the instructions in the README file. You’ll be generating PRs for outdated dependencies in no time at all.
What stopped me?
One thing that has caused me much head scratching, is the Azure Artifacts NuGet feed; a perennial thorn in my side that has led me to compromise my solutions on more than one occasion. The problem being the feed is public and requires authentication, rather than being accessible within a VPN with no authentication (for read access at least). This is obviously only a problem if your repo references packages in such a feed.
I was able to iterate over package dependencies using a slimmed down script from dependabot-script, but I repeatedly received 401 errors. This took a while to solve as Ruby and the Gems ecosystem is not something I’m familiar with. After crudely and repeatedly cutting and pasting code into my script, I eventually realised I was missing credentials for the feed. It sounds obvious, but I’d supplied my Personal Access Token (PAT) for Azure DevOps already, but what I’d failed to realise is that this was just for access to the repo. Applies palm to face. It’s worth noting that Dependabot will use a NuGet.config if it’s available; it will also use any credentials from within the file if your naughty enough to commit them to your repo.
With both my public and private dependencies being checked for new versions, I added the PR creation steps into my script and as if by magic I had 3 new PRs in my repo. One for a private dependency and two for public dependencies. After a fist pump or two and a high-five with the dog, I built a pipeline in Azure DevOps to run the script automatically.
Dependabot provide a Docker image which contains all the necessary components for updating from any or all of their supported package sources. It’s pretty big, and as I’m only concerned about NuGet at present, I took a chance on just running my script within a hosted build agent in Azure DevOps. As it happens, all I was missing was Ruby and that can be added using the UseRubyVersion@0 task — no need to use the Docker image.
And then what?
The next stumbling block was much less obvious than the first, and ultimately required me to submit a PR to the dependabot-core repo itself. The script worked on my development machine, but not on a build agent; the commit relating to the dependency upgrade created by Dependabot was missing the author.email field. It seems the Azure DevOps API is able to deduce my email from my access token, but doesn’t use a placeholder when the Azure DevOps System.AccessToken is used, instead setting the author.email to empty. The problem was exacerbated as Dependabot expects all commits in a repo to have such a field and throws an exception if not.
The folks at Dependabot have been very helpful and suggested I was missing theauthor_details argument from the PullRequestCreator step in the script. They were spot on — I was — but supplying it didn’t help. After more Ruby hacking, I discovered the author_details were being cascaded to the GitHub and GitLab providers, but not to the Azure provider.
I forked the dependabot-core repo, pushing a very crude fix to aid the discussion. Again, not helped by Ruby being absent from my CV, but a few pushes later and a tidy-up from a helpful Grey Baker at Dependabot resulted in a PR ready to be accepted bar the obligatory signing over of IP.
Grey also pointed out that I could test the Gems created by my PR with a small tweak to the Gemfile.
I don’t know if this is part of the Ruby ecosystem, or workflow created by Dependabot, but it’s incredibly useful and the net result was that I had 3 PRs all with the author.name and author.email as supplied in the script; and last but not least, a build had been triggered for all 3.
To be truthful, this isn’t exactly what happened; it took several iterations of the pipeline to discover that the account running the pipeline was short of a couple of permissions. The dependabot-core Gems aren’t that helpful in this respect; failing silently. To cut a long story short, I needed to add Create branch and Contribute to pull requests permissions to the Project Collection Build Service account running the pipeline. Finally I’m seeing the PRs created by Dependabot, and a build running for each to catch any regression issues.
What now?
- The script above only targets a single repo; it makes perfect sense to extend this to multiple repos. This will be my first task on Monday morning.
- I’ve previously contributed towards a consolidation feature in NuKeeper; I’d like to investigate if such a feature exists in Dependabot and potentially contribute if not. I find this useful for related packages that are published with the same build cadence. This would solve the problem where 2 related packages generate 2 PRs, both of which fail individually, but succeed when combined.
- I’d like to investigate a workflow involving Pre-release packages; this would help me confirm downstream consumers are happy with knock-on changes by creating PRs along a dependency chain.
Summary
We have learnt how to integrate Dependabot directly into Azure DevOps, adding to an existing Pull Request workflow to automatically create PRs and run tests whenever a new version of an upstream dependency is published. If you don’t have something similar to Dependabot already, set it up and make life easier for you and your fellow developers.
If you have any comments or suggestions I would be grateful to hear them.
