Following Up on the Industrial-Strength Reality of our Favorite Toolset
A GNET Slide That I Plan to Steal.
Last week’s post about Power Pivot jobs at GNET really drew a lot of interest. I was thrilled to see so many resumes come in. A number of people were/are even willing to relocate across the country! We swamped “poor” Neelesh, heh heh. Good stuff folks.
But I also got a lot of questions – about their methodology, how to prepare oneself to play the “BIM” role in that system, how to get your own organization to see the promise of this approach, etc.
So I thought I’d interview Neelesh for the blog. He accepted, and here we are
Confirmation like this almost makes me jealous
True story: In college, I (Rob) majored in Computer Science, Math, and Philosophy. Trust me, that sounds more impressive than it was – I “gamed” the university system quite well and took a lighter course load overall than any of my friends. (I graduated Magna Cum Lazy.)
So I wrote a lot of Philosophy papers while my more responsible peers were learning to program in C++. Which was fine, because I was a lot better at Philosophy papers than I was at programming.
In a few of those papers, I thought I “invented” a new concept that was groundbreaking. Something that would shake the world of Philosophy to its very core. Old men in tweed suits were going to carry me around on their shoulders chanting my name. But then, back in reality, I discovered, a few weeks later, that some well-known philosopher had written about precisely that same thing decades ago. Happened more than once.
That burned me up. I chuckle, in hindsight, about how angry it made me to be “scooped” like that. One time, a wise professor advised me to view moments like that as “confirmation” rather than invalidation of my desire to be first. I laughed him off, thought he was patronizing me.
During the Q&A with Neelesh below, I realized I might be “cured” of that naïve problem. Sometimes it was like my own words were coming out of his mouth. But now it was exciting, thrilling. My wise old professor had been serious, and correct. It IS confirmation! And it feels good. I was just too young and brash to understand that back in the day.
MUCH more to it than confirmation though…
On multiple occasions during the phone interview I found myself yelling “THAT’S AWESOME!!!” See if you can spot those places…
ROB: You’re a “traditional” BI pro who has very much “seen the light” in my opinion. But that transformation couldn’t have happened all at once. When did you first start thinking that Power Pivot was going to transform the industry?
NEELESH: Yeah, definitely not all at once. I remember first seeing Donald Farmer’s videos about “Gemini” in August of 2009. And I was puzzled about the strategy – here was Analysis Services modeling being performed in Excel! Did Microsoft expect all of us BI pros to stop using Visual Studio and start using Excel instead? It just seemed weird.
But then you spoke at our user group in February 2010 and made it clear that Power Pivot was aimed at the Excel pro. That made a lot more sense to me, and it got me thinking.
Like all BI firms, at GNET we had been preaching the IT-driven BI methodology for years. “Go talk to the business, collect requirements, and then build the DW first. After a few weeks if not months, deliver the very first report.”
This a real challenge, because not only are you asking business to wait more but there is an false assumption that ALL requirements are well defined to build a perfect data warehouse. We BI professionals know all too well, that it takes a lot of time and money to build out the “bottom up Data warehouse, first!” approach, and the idea that requirements will match data realities is a fallacy!”
ROB: “Fallacy.” That’s a great word. What kind of fallacy?
NEELESH: There are three realities of data that “bottleneck” every traditional BI project. One is that data is never actually stored in the manner that everyone thinks it is, so there’s always a lot more re-organizing and re-shaping required than what anyone expects.
The second is that everyone assumes the business is static when in reality it is ALWAYS changing. The requirements you lay down today are always outdated, to varying degrees, by the time the project is “complete” some number of months later. It’s hard to hit a moving target with a glacier.
The third is probably even more problematic – people are just inherently bad at understanding and communicating what it is that they actually need. The first time they start seeing reports, they start asking all kinds of questions and requesting multiple changes, even though you’ve built for them precisely what they specified.
ROB: That third one has been a key point for me too. I call it the “dark matter” of BI projects. So the problems with BI were known to you in early 2010, and now that you understood Power Pivot was aimed at the Excel pro…
NEELESH: I started to realize that Power Pivot could potentially address those problems. It didn’t take us long to verify that it does, once we started applying it. The key point isn’t even about technology really, it’s about people.
The whole point of BI is to change behaviors – stop doing things that hurt the bottom line, and start doing new things that help. So in order to be effective, BI has to reach everyone whose behavior impacts the business.
BI Obviously Must Inform the Business.
But it Fails to Do That Unless the “The Biz” Informs BI!
“Reach” is a two-way street though. It’s not just publishing to every important corner of the business, but factoring in their thinking as well. The “front line” is where everything happens, and out of necessity, there’s a lot of business expertise on the front lines that IT cannot possibly assimilate. That expertise needs to drive the BI system even more than vice versa.
ROB: I agree 100%. That is the magic of Power Pivot that I see with my clients when their BIM/Excel Pros start leveraging Power Pivot, but we tend to be hired directly by the business rather than IT. So I’m still curious how a firm like GNET does it, since I expect IT is usually the one hiring you.
NEELESH: We sit down with an expert from the business, with Power Pivot open, and just start trying to answer some of their questions. In this step we’re building reports right there in the room with them, during the first meeting!
Of course we immediately start running into those places where the data isn’t stored in a way that supports answering their questions. That’s good news though, since in the “old” way, those gaps sometimes wouldn’t be visible until months of ETL/DW work had been poured in concrete!
NEELESH: At the end of the day, the DW is still absolutely needed. It must be built. But we definitely no longer build it first.
When we hold show up to those first “working” meetings with the business, we usually already have built an ODS – Operational Data Store. It’s really just a “one stop shop” of existing data – a single SQL database that’s an amalgamation of multiple different data sources from around the company.
We make zero transformations when we’re loading the ODS. We just pump the data in, unmodified from its original source. The idea is just to have everything at our fingertips for the working Power Pivot sessions.
As we iterate with the business – either within a single session or across sessions – we THEN rapidly make changes in the ODS to support our Power Pivot modeling. This evolution eventually forms the design for what the “real” DW should look like when we do build it.
ROB: That sounds precisely like what we’ve been doing with databases from the beginning. Of course, we didn’t have a DW “pedigree.” We just started using databases in concert with Power Pivot, in a purely pragmatic, “address whatever needs have emerged today in our modeling efforts” manner. I knew we weren’t doing “real” DW work but didn’t know what to call it. I started calling them “Data Marts” but I suppose now I would consider calling them ODS’s. Cool.
NEELESH: In our experience, 90% of organizations heavily use ODS’s, even if they already have a DW. The ODS arises naturally in response to limitations and failures of their traditionally-designed DW. A lot of times, people start thinking the ODS is the DW, which remains hidden behind the scenes! So a supposedly “legitimate” BI solution driven by a DW devolves into an ODS with hundreds of one-off reports built against it.
So the ODS practice is widespread. It’s just a natural human response to the problems. In many ways you are lucky you approached this fresh, with Power Pivot, and never went through the old way.
ROB: I knew it! But I did suffer the old way once, with the football project. That experience, and the contrast vs. the ODS / Power Pivot approach when I re-built it, really made me comfortable with our “no formal DW” approach. I never doubted its legitimacy as a problem solving approach. But… confirmation still feels good.
…Part Two of this interview is now live here.