You Can't Sticker Your Way to a Database
Slapping Data Governance, Stewardship, and a "Data Catalog" (really a Data Dictionary) onto data structures no strong data person ever designed, because: "files", is the same as slapping metadata on files and expecting the benefits of a database just because you write SQL on top of it.
Every few years we try to rebuild the "database". First it was MapReduce, now it is metadata on files.
I remember a few years ago, I was working at a startup. The database went down, and the CEO bellowed "Can't we just turn this into files, so it never goes down?"
In the end, the "files" guys won, which they are pretty happy about, because Data Modelers/Architects always slowed them down, in their opinion.
Witness the decline of the data people, now the “files” guys want the Data Governance people to fix it.
Because, in the end, AI does not understand what was built.
Slapping Data Governance, Stewardship, and a "Data Catalog" (really a Data Dictionary) onto data structures no strong data person ever designed, because: "files", is the same as slapping metadata on files and expecting the benefits of a database just because you write SQL on top of it.
Tell me why I'm wrong.
We had to do some of what has happened because of the profusion of data sources, especially unstructured data, and effective use of Data Producers and Consumers and parallelism.
What did we lose, and what are we continually working around?
We got away with "metadata on files" for one reason: experienced data people silently supplied the meaning the structure didn't. That was the hidden subsidy. Hand that same pile of files to an agent, and the subsidy is gone. The agent reads a valid catalog and still gets the meaning wrong, because a Data Catalog that's really a Data Dictionary tells it what a field is called, not what it's for. And a guessing agent doesn't fail loudly. It fails plausibly: confident, well-formatted, and wrong three systems downstream.
The answer isn't to un-build the lake or fire the "files" guys. They solved a real problem with unstructured data and scale. The answer is to put the modeling discipline back above the physical layer: a governed semantic model of what the entities mean, independent of whether they live in files, tables, or a catalog.
That's the layer an agent can actually trust.