Do you remember in The Last Crusade, when Donovan drinks from the wrong Holy Grail and proceeds to age all the way to dust in the course of about 10 seconds? The constant torrent of newly released products and functionality from the biggest cloud platform players and all of the toolsets that interact with them sometimes has me imagining any cloud solution as a potential Donovan. The question is really what we’re supposed to do about it.
This week, I found myself looking at the recent AWS update to their Well Architected Framework Data & Analytics Lens and it got me to thinking about what, within Architecture, we could do to maintain the flexibility that we would need to easily pivot as things changed and the “right” architecture became different.
Even just looking at a subset of the table of context is a thought exercise. There are pieces that making me consider all of the various projects that I’ve worked on in the past and what I would do given the chance to re-do them now. You can put everything in place, but the right choice of a solution that is the best-performing or most cost effective is going to change over time.
So when something new comes out, like Microsoft Fabric, what does the new choice look like? The long-term benefits of whatever the new solution provides the business are always going to be the biggest factor in deciding when to make a change. That said, the scope and complexity of how to go from current state to wherever you want to head is tied heavily to how the original solution was architected.
For AWS and Azure, I go back to the philosophies that they seem to build their solutions around. I think it’s easy to argue that Azure thrives on building tighter integrations to enable easier and more seamless end user experiences across different parts of a solution while AWS is attempting to enable loose coupling seemingly trying to apply a Microservices approach wherever possible. There are definitely tradeoffs at each end of the spectrum.
However, when I come back to the desire for more flexibility, the smaller the individual pieces of a solution are, and the more loosely coupled they are, the more easily someone would be able to develop, test and successfully migrate a replacement that moves the overall solution forward. That doesn’t necessarily mean choosing one tool or cloud over another. What it does mean is trying to design in a way that is somewhat modular and with less dependencies. The more the number of dependencies, the more sequential and less scalable a job is, and the longer the chain of those tight dependencies, the harder it is to trade out any one piece without affecting everything else. That ability to consider external alternatives also goes up with non-product specific integration points such as API’s. Much of the application development world has learned to design this way, but I think that the data world could do a better job.
What I’m suggesting isn’t a drastic change. Every design of these solutions has elements of both art and science and all I’m asking is that a thoughtful approach is taken when considering what you’re going to do. Not all of the decision on the design should be around what it needs to do today. We should definitely be thinking about the possibility that the solution is going to age before our eyes, and we’ll be back in this design phase before we know it.
Finding Order in the Chaos 