You Can't Sticker Your Way to a Database

Every few years we try to rebuild the "database". First it was MapReduce, now it is metadata on files.

I remember a few years ago, I was working at a startup. The database went down, and the CEO bellowed "Can't we just turn this into files, so it never goes down?"

In the end, the "files" guys won, which they are pretty happy about, because Data Modelers/Architects always slowed them down, in their opinion.

Witness the decline of the data people, now the “files” guys want the Data Governance people to fix it.

Because, in the end, AI does not understand what was built.

Slapping Data Governance, Stewardship, and a "Data Catalog" (really a Data Dictionary) onto data structures no strong data person ever designed, because: "files", is the same as slapping metadata on files and expecting the benefits of a database just because you write SQL on top of it.

Tell me why I'm wrong.

We had to do some of what has happened because of the profusion of data sources, especially unstructured data, and effective use of Data Producers and Consumers and parallelism.

What did we lose, and what are we continually working around?

We got away with "metadata on files" for one reason: experienced data people silently supplied the meaning the structure didn't. That was the hidden subsidy. Hand that same pile of files to an agent, and the subsidy is gone. The agent reads a valid catalog and still gets the meaning wrong, because a Data Catalog that's really a Data Dictionary tells it what a field is called, not what it's for. And a guessing agent doesn't fail loudly. It fails plausibly: confident, well-formatted, and wrong three systems downstream.

The answer isn't to un-build the lake or fire the "files" guys. They solved a real problem with unstructured data and scale. The answer is to put the modeling discipline back above the physical layer: a governed semantic model of what the entities mean, independent of whether they live in files, tables, or a catalog.

That's the layer an agent can actually trust.

Nobody Reads Anyone Else's Code — And Now Nobody Reads AI's Either

Cheryl Dopp

• You

In a world of AI, having a unique voice and clear, personal vision will be the differentiator.

16m •

I enjoyed reading the post by Andreas Horn yesterday, and I just want to point out how many of the comments were developers stating that they aren't even reading the code before shipping it lol.

That dovetails with another comment I made about how developers and project teams are reluctant to re-leverage what has been built, and to refactor or extend existing codebases. In practical terms, I've almost never witnessed a developer reuse another developer's or team's code.

The argument is always about risk (risk of dependencies intra-project) and cost (it takes time to read another's code). They'd always rather write it from scratch, even if it is largely replicating existing functionality in another set of code.

This introduces the same problem that duplicated data produces: different and contradictory logic in different places and the overhead of resolving discrepancies between the codebases or data stores.

And the cost of pulling the casino lever to regenerate something new seems cheap, at the beginning.

I don't think these two problems get any better when AI is writing the code, and am reminded of it after reading the comments.

Do you?