A Frontend Engineerâs Stroll Through DevOps | by AFreeChameleon | Mar, 2025
6 days ago
I donât love programming, I love building things
This is my mantra for why Iâm in the software industry, I rarely find happiness in making use out of all features a language has to offer, nor do I really care about the myriad of tools and languages I use on my projects. The dopamine derived from saving a few kilobytes of memory doesnât itch my scratch, but it is making something and using it to help me in life, thatâs what gets me out of bed in the morning, and working on personal projects late into the night.
From the idea of building things, comes a wide knowledge of each part of the system you are making. From designing to deployment Iâll know how to do it all, and itâs this knowledge that itches my scratch in this industry.
Unfortunately, jobs are seldom like this. My first programming gig was a front end engineer where I was to be a machine, with my only purpose being converting a .psd file to a .html file, letting someone else handle the backend responses and deployment (usually two different teams). This was a far cry from who I wanted to be in my work life, my fulfilment and pride doesnât come from being a machine designed to replicate one task ad infinitum. However my next job was exactly the same, and the next one, and the one Iâm at right now. The vast depth of accumulated knowledge in this one topic with no other areas to fall back on became my worst nightmare:
All of my eggs are in one basket.
For a year I kept pestering my manager for anything to do other than frontend with no luck. My hands were tied. All I could do was express myself through personal projects which even though they never brought anyone else joy, they made me happy as I was scratching my itch.
One year later I got a notification for a meeting called high availability. Searching through my pile of eggs, I couldnât find anything around what that meant so curiously I entered the meeting. Upon joining, I found the devops guys, me, a colleague, my boss and his boss. âA client wants us to make our platform highly resilientâ my line manager said in a cautious tone. âyouâre going to work with kubernetes to make this a reality for themâ.
Truly a frabjous day indeed. Now I still had nagging prejudicial thoughts about kubernetes being contemporary with early budding startups, used as a tool less for scalable environments and more for extracting money from investors, however, I couldnât ignore the excitement built up inside, so I donned my brand new devops cap over the faded, ratty frontend engineer one that rested on my head for the whole of my career.
We need to talk about our stack first to give you context before I delve into my journey. The company I work for who I’ll call Boop Ltd. Came into existence in the mid 2010s. And sort of similar to how you can tell an outfit is from the 2000s, you can definitely tell our stack was from the mid 2010s. Our NodeJS+express backend was the denim jacket, paired with the MongoDB sport sunglasses and a microservice for every spike in our gelled hair, what we’d call a relic of a stack today was the bleeding edge of yesteryear.
But in 2013, we didn’t know what a gruesome, slippery hole we’ve dug until 11 years later. Technologies become outdated as soon as they’ve scrambled into fashion, like take a look at all those OOP languages a few decades ago, they were the hottest thing since the sun and what a mistake programming in those languages were (Zing!). And now our webapps’ performance and resilience was in the hands of the sluggish, hamfisted hands of Javascript. The devops team were going to have to work wonders to make up for our sins, so they called in a couple of frontend devs with no docker & kubernetes knowledge (me and my colleague!!) to take this mastodon of a problem down so a big client will give us big money.
Yeah I’ve been begging and pleading my manager for a morsel of something other than frontend for a while, but little did we both know, I ended up having to lead the devops project where if the client’s systems go down for a mere 9 hours out of the WHOLE YEAR (8760 hours) we lose them and their big money. Not my choice for my first foray into the k8 cookie, but the k8 cookie doesn’t choose how to crumble so into the fray I went.
The first job on the docket was one of the simplest and yet one of the most crucial, right now, If our NodeJS backend (yucky) fails to connect to MongoDB (ew), it will just hang and break until the client’s sys admin manually restarts it (gross). On kubernetes, dammit! Such a simple solution to this and yet noone bothered, we just used liveness probes to ping our vital services every minute and if anything’s broken, a restart is triggered. Super easy.
However, an omen of the future of this project came after I finished this work. The devops team got reassigned to another project. So Iâll be picking up the slack theyâve left behind, and it was a lot of slack indeed.
For some extra context, we donât just have MongoDB to store all of our customerâs information. Later on in our productâs lifecycle we found out itâs incredibly slow for rapid changes (who wouldâve thunk it), which is where we brought in Opensearch (Elasticsearch in an open source wrapper). While it improved speed, this has been the biggest headache Iâve ever ran into and itâs all our fault.
Documentation
The most important part of programming for big corporates is documentation. Every environment is different so every company needs to write steps to use/setup their software in a format even a child could follow. As I glanced through our documentation, all I could see were erratic scrawls tangentially related to the topic. Words without meaning to anyone outside this devops cult, areas where you can tell they missed a step or forks in the road with the author only hoping youâll know we use AWS and not Azure. I could tell there was going to be long dark days ahead but didnât quite understand the gravity of the peril me and my companion were being thrown into.
The first lesson I learned was the map to success was fragmented across the entire team, each person giving their piece of the puzzle in hopes I can match it up and do something as simple as perform a backup and restore. With my trusty Obsidian, I scribbled each oracleâs instructions. bewildered at how only one knows how to backup our mongo and how only another knows how to backup opensearch.
After days of scrambling together a better, rewritten plan than the jumbled slivers of knowledge that was handed to me, I did a backup & restore of mongo⦠And it completely broke everything. It turns out the only testing this team did was just a small glance of if there was a hint of the old data in our databases. No testing the product to see if it rips itself to shreds which it has just done before my eyes.
A What Request?
Our devops team is the picture in your mind when you think of a 1990s webmaster. one person festering in a nest of server racks, ethernet cables like vines entangling all who enter. From one look your tech mind tells you if this solitary character left for greener pastures, the company would explode.
In todayâs world these people tend to not exist anymore because these people did leave, and companies did explode and any onlooking businesses wrote notes on how not to be like that company. Not our company. For this team has both the incredible knowledge of the 1990s webmaster and the devastating disorganisation. 2500 line merges straight into master, ruining our releases multiple times with no higher power to tell them to stop.
I got a taste of that end of the stick when I tried to do a simple redeployment and it failed miserably. Thatâs odd I thought, it was working yesterday and Iâm using the same release branch. What happened? Crawling through the stack traces, turned into a company wide witch hunt, knocking on peopleâs doors and asking them if they killed the release branch. My search ended up on the door of the devops team lead who told me they messed up the docker image name, so all deployments are now broken. The dimly lit clock of my MacBook shone 4:50PM and so I just fixed the problem and clocked out.
Turns out the backups did work, just after a week of trial and errors. With every error leading to out of sync data. If our £2m client had their data corrupted from a terrible backup process, they would promptly rip up the contract and tell us to beat it, so I made myself make a script to perform every fragile instruction in this china shop of an environment so we can stop trusting the client to follow our instructions to the letter and not to break things because I wonât be around to yell noooo before they break things.
A few weeks into moving over to devops, I got several hints at why this department had these flaws, after unearthing enough of this cursed knowledge, what lied underneath was a tangled web of questions for me. From other, often very talented engineers asking about how to do deployments, and build container images and the like, but this goes to show how much of a black box our devops process is. And as an ex-frontend dev who knew little about our backend and nothing about our devops, it was now my responsibility to share what Iâve learned with my comrades.
What good would all of this nonsense be if the customer couldnât see it. The next chapter of this project was about showing off what we actually did in a shiny dashboard using Grafana.
Iâm not fond of companies that have their own query languages, an api in a language the world knows is far better than anything a team of devs can cook up in a sprint. And also itâs so annoying to learn something specific to a company, like a mechanism that you send one employee into who gets taught how to use it and out comes one employee that canât be fired. I guess thatâs me for nowâ¦
Learning promql (Prometheus⦠QL) along with Amazonâs cloudwatch language was reasonably easy for the most part, just a bit restrictive on what I could do, but Iâve managed to get a dashboard hacked together with my new found, âunfire-ableâ knowledge.
A favourite speaker of mine, Kevlin Henney did a talk about agility not being the same as velocity and why we should favour the former over the latter. A talk not many have heard apparently, as it is routine for corporate projects to dart around in different places, scopes to expand and expand rather than vacuum seal the features we need. Velocity means nothing if weâre going in the wrong direction, and with each link of the daisy chain above me getting assigned to other projects, and all these sinister discoveries Iâve made like woodlice under a rock. After a while, proceeding my initial fear was an obligation to clean up.
So thatâs what I did, if no-one was going to decide what the scope of the project was, I was going to, whether the daisy chain likes it or not. After all, I rewrote our lackluster documentation, every scrap of information hard fought for to be given to the rest of the company on my own time, automating what shouldâve been years ago.
While I may seem like Iâm self aggrandising, I was most disappointed of all at myself. Like a tiny leaf, blown about by the breath of my superiors, like a flimsy twig bending but not breaking under the feet of those above me, I was assigned a monumental task, with everyone in front of me one by one, handing me their slack, until I looked up and no-one was there except a balloon of features for a demanding client. Iâm nearing the end of this project and Iâve learned that hard working employees just get more work. More tasks are given to them. While their salary depreciates under the sedimentary layers of inflation. Iâm not going to purposely underachieve, but rather respect myself for the hard work I do put in, with hope that that self respect will come in the form of telling people to go away when I have too much on my plate, and maybe asking for more pay. A lot easier said than done though.
And in the end, I really enjoyed it. Bjarne Stroustrup said:
There are only two kinds of languages: the ones people complain about and the ones nobody uses.
And this is true for a vast array of things as well as languages. I loved having to chop through the vines to get what I want and clear a path for others following in my footsteps because this is how you learn. Not following instructions or being told how to do things a certain way without an explanation, but making mistakes and improving on them to make the ecosystem better. And I wouldnât have learnt about our tech as much if I followed a paved path rather than made it.
Before you go:
