Hello! I’m George Dutton, Senior Data Engineer Managed Services, I’ve been with Coeo for a little over a year now, Read more here!, I was asked to be part of a 3 month secondment on to a data engineering project, building a central HR solution in Microsoft Fabric. Before, during and after the project, I found myself reflective, what will I learn, what can I bring, what did I need to prove to myself. Having been in Managed Services for 6 years now, I’ve only worked on enhancements, Read an example here, but this was my first full green field project, and I was nervous, but I learnt a great deal. These are some of the lessons I learnt along the way, I hope to bring to back to Managed Services and my career at Coeo.
Don’t let perfect be the enemy of good
Coming from Managed Services, for my first full development gig, I wanted everything to be perfect, ideal, never failing, no bugs, amazing, A* interview answer. Reality soon set in and made me realise that was a pipe dream.
I couldn’t think of every use case, sneaky bug, corner case, surprise that came my way. Perfect code doesn’t exist, I hate to admit. I had to accept I could still deliver good, solid, reliable code but the bugs will come, I had to embrace them. The code changes had to happen, I had to realise that my code wasn’t perfect, and not be precious over it. I could always do better, but perfect is often the enemy of good.
Products and tools are not perfect either. Outages will happen, copy activities will rerun, assumptions will be tested. I tried to add as many safeguards to my code, hoping the support team would thank me one day. It turned out, I was the thankful one. When ingesting all historical data, the controls I had written were going off. Suddenly “that will never happen” turned into “wow that just happened”. I learnt, ‘Don’t assume, code protections for it’.
If you want to go fast, go alone. If you want to go far, go together.
I built a great connection with the client, much greater than before, working shoulder to shoulder with their engineers and architects led me to build bonds with them and I will miss working with them, as they are a great bunch of folks. Every personal connection I built, I found paid off for me in the long run. When we had to make difficult decisions, explain delays, overcome hurdles, the trust they had made those moments much easier to navigate. I didn’t appreciate the Copilot notes outing me with ‘personal anecdotes were shared, such as George’s visit to Twickenham’ but I enjoyed playing the ‘what could I get Copilot to note’ game!
I really enjoyed working in a small delivery team as well, a principal, a project manager and me made a small band, each with our skills we could bring to the table. I found I work better bouncing ideas of others, I shouldn’t expect myself to have all the answers even if I want me to. Learning where I was strong and where I was weak was a huge lesson for me during the project, one I will try and keep going forward.
A stitch in time, saves 9
This is a phrase my grandma used, and it couldn’t be more relevant to my time in consulting. Decisions made earlier, would come back to haunt me, some came to save me. Those early design decisions are so critical to get right, they were worth the time spent on them. Others, in retrospect, I wish I had dwelled on more. One specific example was discussions around source systems hard deleting data. I recall the meeting where we said we would come and revisit it, well that revisit was thrust upon us during testing due to data total discrepancies. The client learnt something new about their source system, I learnt something I’ll never forget, “Listen to my gut” and make sure to follow up on the easily forgotten ‘let’s circle back’ moments.
The faintest ink is better than the best memory
Documentation is often the most important non-functional requirement. Why? Because it matters. Even if I thought I’d be on the project forever, I know I would forget something down the line. Why did I pick X over Y pipeline design, my memory fails me. “Write it down” I found myself screaming to my past self, even now. Signed design docs, linage files, architecture review emails saved me too many times to count. Your future self will cherish your documentation, your replacement will thank you, Managed Services will worship you.
AI can help you, but sharpen your own pencil
‘Vibe coding’ is rife at the moment. Would I suggest it, no! Did AI write some of my code? Yes! The blank SSMS query window was scary, I’m used to debugging 1000s of lines of SQL, Python, PySpark… How did I replicate that in Consulting with 0 lines written? Gen-AI! It could spit out 10,000 lines of SQL in seconds, brilliant, I had something to debug. AI is great… it’s not magic, sharpen your red pencil you’ll need it. I spent less of my time on code written many times before, and spent the majority of my time on specific code for the client.
Every consulting project is different, AI will not know your client’s systems well, they won’t know the data anomalies hanging just round that corner, but, put to the right use, it can save you time writing the ‘boring’ bits. Spend what precious time you have in this world on the exciting bits!
Rubber ducking really works
During SQLBits, I seem to gain a rubber duck for my desk. It came in handy on quite a few occasions. When I ran out of AI credits, I turned to talking the code through with my rubber duck, my cat, my fiancée, my colleagues. This is a phenomenon because it works. I had a colleague that talked to action figures… and I’d argue his code was the cleanest I’ve seen. Don’t knock it until you’ve tried it. I will now be explaining my incidents to the rubber duck, I’m sure it will keep listening to my ramblings for many years to come.
All Good Things Must Come to an End
After 3 months secondment, my time on the project had to come to an end. I really enjoyed working with the client, speaking to them most days, working through their data and their problems. It was a challenge for me, one that I was reluctant to accept at first but I look back on fondly, especially how much I learnt about myself, data consulting and Fabric. The project has moved into phase 2, with a new consultant who I trust to add, improve and develop with the client further, I wish them the same enjoyable journey I had. One final proverb, ‘This too shall pass’.
Summary
From just a short stint in a project, I felt I learnt a great deal about myself, delivery, and life lessons. For those who skipped to the end, here is a TLDR:
- Don’t let perfect be the enemy of good: Managed Services side of me wished for perfect, I had to accept good. Products aren’t perfect, assume they will let you down.
- If you want to go fast, go alone. If you want to go far, go together: Building connections helps you in the long run. Surrounding yourself with a great team gives you a chance to succeed.
- A stitch in time, saves 9: Early decisions are the hardest to undo down the road. Build for success from day 1, don’t put it off or ignore your inner voice.
- The faintest ink is better than the best memory: Documentation will save you, the client, Managed Services. It’s worth it’s weight in gold.
- AI can help you, but sharpen your own pencil: AI can do the easy noddy stuff for you, leaving you with the fun, homework checking, correcting, build custom areas to you.
- Rubberducking really works: Don’t knock it until you’ve tried it!
- All Good Things Must Come to an End: I’ve loved my time on the project, the ups, downs, mysteries and building something from scratch.
I’ve learnt a great deal on this journey, on to my next adventure!