To Scott Taylor - my data story about the Meta Grid

The preface of my book Fundamentals of Metadata Management

How metadata management is perceived (to the left) and how it is performed (right)

Scott Taylor asked me to send my data story on the Meta Grid, so he could comment on it, and - I’m thankful Scott! Here it is: The following is the current preface of my book Fundamentals of Metadata Management, to be published with O’Reilly in 2025. It contains two paragraphs, About this book and Why I wrote this book. Everyone, including YOU, can comment below. I read all comments. You can hear me read this post out loud, on in this episode of my podcast special series The Meta Grid Notes.

About this book

Something crucial is often overlooked in how companies approach metadata management. 

This book addresses that missing piece. 

Metadata management relies heavily on repositories, each serving a distinct role in bringing companies into control of their IT landscape. However, few companies have a holistic approach to this critical aspect of Metadata Management. Metadata repositories are often managed throughout various teams in organizations, leading to high costs, risks, and confusion. 

During strategic initiatives, companies often encounter a recurring pattern. Whether it's a merger/acquisition, a transition to cloud computing, enhancing data privacy, or implementing a durable data science project, certain questions persistently arise: What applications do we have? Who owns them? What data do we possess, and how is it utilized in our processes? Which technologies support what capabilities? Where are our servers located? 

In most companies, management teams ask these questions and realize that no one can really answer them - or, not completely. As a result, they hire a team of external consultants that help them create the overview of the IT landscape they need. This snapshot is typically stored in a metadata repository and sporadically maintained by internal employees over time. Over the years, this process repeats itself again and again. It’s not a resilient approach, and this book offers a way out of this recurring cycle by: 

• providing a (non-exhaustive) overview of metadata repositories (Part I)

• describing a team that coordinates metadata repositories (Part II)

• suggesting a meta grid architecture to make metadata repositories robust (Part III)

Why I wrote this book

In most companies with more than a thousand employees, there is often a lack of understanding of the IT landscape. 

Instead, certain people know certain parts of the IT landscape.

This is a catastrophic reality. 

My first job was in big pharma. I remember this chemist that I worked with. A brilliant guy, an industrial researcher with a PhD. He had thick glasses, a weird, satanic smile, and a machine-gun kind of laugh. He used to make bombs in his spare time, just for the fun of it. One day, he showed me a long, long list of something called the NITSO - it was the IT Systems Overview. I was 20-something years old, and I just stared at the list of IT Systems that went on and on. Thousand of systems. I asked him what all those systems were used for and then he grinned at me and said that no one really knew. I remember my surprise when he told me that a large portion of the systems probably didn’t do anything at all. But it was just too dangerous to turn them off, he said. Because no one knew what would happen. Maybe nothing. Maybe a factory would shut down. Nobody knew. Or at least, nobody knew, if anybody knew. So instead, he continued, we kept paying the vendors of the systems. And then he laughed his machine-gun laugh.

I didn’t know what to say.

As I progressed in my career and worked for several companies, I learned that this is more or less the reality everywhere: Companies don’t really know their applications, they don’t really know how they are integrated, they don’t know what data is actually in those applications, and if they are getting what they pay for, they are paying for the same kind of thing twice - or more! - because they buy multiple, basically identical systems. 

Over the years I learned how to accept that reality and fight it, as a specialist, leader, and enterprise architect, focused on data. You can only fight that reality very, very slowly, gradually expanding your knowledge. At least I thought so.

One day I had a simple yet powerful idea which forms the core of  this book —coordination of metadata repositories. 

See, the big problem with not having an overview of the IT landscape is not that companies don’t possess this overview. The problem is that they possess too many overviews. These are called metadata repositories, and they are all deeply technical, very different in scope and yet look at the same thing, namely: the IT Landscape of a company. And the horror of it is that they never match.

But you have the power to change this - and the rewards for doing this are substantial!

The reason I wrote this book is to provide a pragmatic approach for gaining a comprehensive overview of the IT landscape within a company. By understanding, grouping, aligning, and exposing all the metadata repositories of your company, you can realistically achieve this goal, with returns including:

• a smoothly running IT landscape

• significant cost reduction

• better use of data for innovation

• enhanced data privacy

• enhanced data security

• a greener IT landscape

• and basically, just a company that knows its IT landscape

Understanding the basics of metadata and its management through repositories enables you to construct a more resilient and enduring overview of your IT landscape.

---

Over to you, Scott - What’s my data story?