Turning Frustration into Fuel in Data Engineering
Frustration is more than just an unpleasant emotion — it’s that feeling of helplessness when you can’t change or achieve something important [1]. Most often, it strikes right after a good idea stalls, a plan falls apart, or a goal becomes unreachable.
In the data world, this is a familiar feeling. At Data S2, we’ve lost countless hours to repetitive tasks, unnecessary meetings, false emergencies, or so-called “miracle” tools that promised to solve all our problems. In the end, it all came down to one word: frustration. But what if frustration wasn’t the end — what if it was the beginning?
Frustration as fuel
If you’re building something that truly matters, you need to learn how to use frustration as fuel — creatively and operationally. This mindset shift changes everything. Instead of stopping you, it propels you forward. The real question is: how do we apply this to the data lifecycle?
The improvised way (you probably know it)
The most common solution is also the most dangerous:
"Start now. Fix it later — if there’s time."
No matter your level of experience, this phrase will eventually catch up with you. Maybe you’ll inherit a legacy codebase full of hacks, or a project with zero documentation, or get a last-minute request — like Tolkien’s “last light of Durin’s Day” [2]. This mindset leads to technical debt, lost productivity, and yes — more frustration.
Engineering as the antidote
To break the cycle, we need to apply solid engineering principles to everyday data work. At Data S2, here are the ones we use most:
Configuration as Code: define workflows as code to ensure reproducibility and clarity.
Breadcrumbs: track data dependencies as if you're leaving a trail.
Automate anything repetitive: if you do it more than once, automate it.
Fail Fast, Learn Faster: experiment more by reducing the cost of failure.
Modularity: break the system into reusable, testable blocks.
Separation of Concerns: ensure each module has a single responsibility.
DRY (Don’t Repeat Yourself): extract and reuse common logic to avoid duplication.
KISS (Keep It Simple, Stupid): simplicity is sustainable — avoid overengineering.
YAGNI (You Ain’t Gonna Need It): don’t build features until they’re truly needed.
TDD (Test-Driven Development): write tests before the code to ensure quality and clarity.
CI/CD: continuously integrate and deploy — keep things flowing.
From chaos to clarity: our internal roadmap
To organize our project development and turn frustration into progress, we use a clear and concise framework:
Abstract: main takeaways in bullet points.
Situation: the context and the problem — backed by evidence.
Requirements: tools, functions, non-functional needs, and bottlenecks.
Architecture Diagram: the system's high-level view.
Sequence Diagram: the ideal step-by-step flow.
Actions: what needs to be done.
Expected Results: what success looks like.
Opportunities & Improvements: after delivery, what could be improved?
And for every task? Use the STAR method
Even for individual tasks, we rely on the STAR framework (Situation, Task, Action, Result):
Situation: what’s the context or problem?
Task: what needs to be done or achieved?
Action: what steps will be taken?
Result: what is the desired outcome?
This structure reduces ambiguity, aligns expectations, and — most importantly — prevents rework.
Conclusion
Frustration is inevitable. But it doesn’t have to be a burden — it can be a driving force. By applying strong engineering principles and structuring your work with clarity, you can turn chaos into process, and process into value.
If you feel like you're wasting too much time on useless tasks or unnecessary fire drills, maybe it’s time to rethink your approach — and let frustration work for you, not against you.
References
Definition of frustrated by Cambridge Dictionary. Access in July 20, 2025.
The Hobbit: Desolation of Smaug by J.R.R. Tolkien.