Date and Date/Time – Sneaky Data Types!

February 21, 2011

 

                                                    image image

These Two Columns Both Contain Dates, Right? Sure They Do!

Relationships that should work, but aren’t?

Here’s a quick tip, one that I think we’ll all need sooner or later. 

When I got back from last week’s amazing consulting/training trip, the team let me know that we had a problem.  An existing PowerPivot model, one that had been working just fine for a long time, had stopped working when we refreshed the model with the latest data:

image

All the slicers were now indicating that there wasn’t any data.  And those empty rectangles on the right?  Those are charts.  They just weren’t showing any data anymore.

What’s the problem?

We pretty quickly determined that it had something to do with the date relationships.  When we cleared the date slicer, data came back.  So it was just when we were filtering by Date that data disappeared.

imageIn cases like that, the first thing I like to check is data types.  Somehow, did one of the columns get changed from Date to Text?  Nope, not this time.  They both are definitely still Date, as evidenced by this screenshot from the PowerPivot window ribbon, here at right.

 

But Data Type Isn’t Enough!

Notice the Format option there, though?  Try setting both related columns to a format that displays time as well as date:

image

Now look at the two columns:

                                         image   image  

Like I said, sneaky little Date/Time data type.  When we got new data this time, one of the source systems in the loop decided to include the time of day, whereas before, it had been omitting it.

So yeah, the two columns don’t match up anymore even though by default they LOOKED like they matched.  Sneaky.

So…  if you have a relationship on two Date columns, and the relationship does not seem to be working (all of your measures are returning blanks), this is a likely culprit.

Two ways to fix this

The quickest fix is to create a “clean” calc column that strips the time:

   =DATE(YEAR([CalendarDate]),MONTH([CalendarDate]),DAY([CalendarDate]))

Then you use the calc column for the relationships rather than the original.

But the better, more reliable way to clean your date columns is in the underlying SQL (as long as that’s an option).  If your SQL sources always trim your Date columns down to pure Dates, and truncate Time, then you never have to write calc columns for this purpose again – solve it one place, and save yourself a lot of ongoing work.

Furthermore – columns imported from SQL end up being a lot better compressed by PowerPivot than calc columns, AND this often results in faster pivot performance as well.

If you do the trimming via Views, you can keep the time component in your SQL tables for later, in case you want to do “time of day” analyses.  But your default views that you import should always be protected against this.

And remember, if you aren’t a SQL pro yourself, having a good relationship with the folks who run your databases is a very positive thing.  At Pivotstream for example, I never touch SQL.  But my colleagues who maintain SQL are very helpful, and our cooperation lets us do things that otherwise would be impossible.  One of my favorite themes – cooperation between PowerPivot pros and SQL pros – and I will be hammering said theme every chance I get Smile


James Bond, international biz intel operative

February 19, 2011

 
PowerPivot Can Turn You Into James Bond

"OK, wow. THAT is SICK. Just. SICK."
-Lead Analyst at this week’s consulting/training client

"We just eliminated the need for our entire first development sprint in two hours."
-Director of Business Intelligence, same client

“We’ve been talking about how badly we need this particular analysis forever.  I can’t believe it – all this time we had the ability to just DO it, but didn’t know.”
-CEO, same client

 

Quite a Rush

It’s 5:45 am in Atlanta Hartsfield International Airport.  I’ve just flown overnight after two intense days at a client in California.  I have ninety minutes until my next flight.  I have not slept.

And what am I doing?  I’m walking – practically gliding – through the vast tunnels beneath the airport.  For long stretches I am the only human being present.  I have no interest in the shuttles, nor the moving walkways.  I’ve got too much energy to burn off.  I simply feel too GOOD to sit down, or even stand still.

Why do I feel so good?  Take another look at those quotes up top.  Pretty damn cool.  But I suspect that it’s hard to relate to why I was flying so high without a little more background.

And even more importantly, I want you to understand why and how PowerPivot will take you to similar places.

In the company of amazing people

If you read the quotes above, and your reaction is something like “Rob’s client didn’t know what they were doing, and Rob went in there and opened their eyes,” I need to clear some things up.  This is simply an amazing organization, made up of amazing people.

Creative.  Sharp.  Energetic.  Nimble.  Successful.  Irrepressible.  Open minded.  Those are the adjectives that come to mind when I think of these folks.  In an incredibly short time, they’ve built a VERY successful business.  The kind that makes me envious, to be honest.

These guys are rock stars.  They’ve achieved things that in many ways don’t even seem possible.  Movies are made about teams like this – you know, movies like The Social Network.  Or the Pirates of Silicon Valley.

I’ve always had a reasonably positive view of my own capabilities.  But for most of my career I’ve simply been a cog in a monstrous machine.  Only recently have I been experiencing this new vibe – one where I can drop in on a team like this one, sit down with the leadership, and have this kind of impact.  It’s humbling.

Did I say “humbling?”  OK… Forget what I just said Smile

PowerPivot Can Give You a Sense of Power But Not the Dr Evil Kind“It’s humbling” – that’s what you’re supposed to say at times like these.  But it just isn’t true.  Experiences like last week are the opposite of humbling.  “Humble” doesn’t generate a rush.  It doesn’t excite or invigorate.  No, what I was feeling early that morning was…  power.  Pure, amazing power.

But not power in the Dr. Evil, “rule the world” sense.  No, the power available to us here is simply one of significance, of impact.  The ability to affect things – much closer to the definition of “power” in physics than what you find in politics.

I could get used to this… but it’s a brief window in time

Back to the tunnels underneath Atlanta:  I started thinking where this could end up going.  In the next year, as PowerPivot’s reputation spreads further, will I (and people like me) be travelling the world, helping leaders of international businesses around the globe revolutionize their ability to “see” clearly?

It certainly is plausible, and no lie, that’s unbelievably exciting.  But then I came down from my cloud just a little:  this is going to be a brief window of opportunity (to be the teachers), and won’t last for long.  Remember that PowerPivot derives its real power not just from superior technology, but also from its accessibility.  It is something that empowers Excel pros, and will do so for literally millions of people.  We’re just in a transition phase.

To illustrate, imagine if we had no spreadsheets at all today (and no PowerPivot either), and suddenly, out of the blue, someone invented Excel.  Not even Excel 2010 – let’s say this first version had the feature set of Excel 97, and nothing like PowerPivot.  Can you imagine the change that would be in store for the world, as the spreadsheet began to work its way into the business culture for the first time?  Imagine that happening in a world where PC’s were already ubiquitous and turbocharged (like today) – not like the world that spreadsheets first entered (where PC’s were still quite rare, and woefully less powerful).  If you were one of the first to learn this new tech, you would be very much in demand.

That’s kinda how I view where we are.  Naturally, people like us, who are on the leading edge of this wave, we have a huge advantage.  But it’s temporary.  This is where humility returns.  Humility is a good thing – it keeps things in perspective and positions you better for the next round of change.  Our roles, for now, are simply to help spread this revolution.  I’m starting to understand just how BIG the revolution is going to be, and it’s thrilling.  It’s a limited window of change, though, so keep that in mind.

But boy, it’s going to be quite a ride.  For all of us.

Post Script:  A Hat tip to Mike

Who’s Mike?  Mike is the guy who had to sit next to me on my next flight, from Atlanta to Cleveland, and deal with my overly up-tempo and chatty mood (at 7 am).  At one point he suggested I include Dr. Evil in this post.  Which is, of course, about the best advice I could hope for.

Mike was coming to the end of an interesting trip of his own, returning from Brazil.  Mike’s company is one of the few American firms that still actually MAKES things.  Physical things.  And sells them not only in the US, but abroad.  Fascinating conversation, thanks Mike.

Oh, and Mike is a longtime Browns fan who was in the stands for both The Drive and The Fumble.  Mike, you deserve redemption, I hope you get it someday.  And there’s another Mike (Holmgren) who I hope is listening.

While we are on the topic of football…

Hugh Millen:  Longtime Excel Pro, PowerPivot Pro in the MakingAlso last week, on the flight out to the west coast, I was exchanging emails with a guy who used to be a starting quarterback in the NFL.  Did you know that Hugh Millen is an Excel pro?  Of course you did, right?

Anyway, as a former player and current TV/radio analyst, Hugh obviously has a deep understanding of football.  His passion for numerical analysis sets him apart though, and suddenly even someone like me has common ground to discuss with him…  like PivotTable macros.

How cool is that?  Here’s a guy who has enormous “cred” in his field – a field that you don’t typically associate with business intelligence.  It’s not like he NEEDS to be diving into spreadsheets in order to be successful at what he does.  But he knows there is still a lot of additional value he can provide by looking deeper than what his trained eyes alone can tell him.  So he’s been an Excel pro for a long time, probably longer than me, and is now becoming a PowerPivot pro as well.

Moral:  It doesn’t matter what field you are in.  There’s opportunity lurking in the numbers.

More Posts in the Pipeline

Sorry about the lack of posts last week, there was a lot going on.  I already have a short technical post written for later this week, and there’s another in my head about “Excel circa 2002 versus Excel Today” that I hope to get written as well, so stay tuned.


Speaking at Columbus OH User Group Tonight!

February 10, 2011

 
If you live in Central Ohio and want to see my talk from last week’s SQL Saturday, I will be speaking tonight (Thursday the 10th) at the Columbus SQL Server Users Group.  No need to register, just show up.  Free admission, and I am told there is free pizza Smile

Location:

Goodwill Columbus
1331 Edgehill Rd
Columbus, OH 43212
http://maps.google.com/maps?q=1331+Edgehill+RdColumbus,+OH+43212

Time:  6:30 PM to 8:00 PM


Beta program update

February 7, 2011

Sunday night update:  we’ve received a lot of exciting interest and have identified about 15 great beta sites so far, representing a broad cross section of the industry – BI and SQL pros, SharePoint pros, and Excel pros who are “growing up” into BI and SharePoint via PowerPivot.  Great international representation, too.  Still looking for a few more participants, so drop us an email – beta@pivotstream.com

What beta, you ask?  Click here to see last week’s announcement.


Is the future of Office too tightly tied to SharePoint?

February 7, 2011

Dick Moffat wistfully mourns the Office Developer Conference...  and the abacus :)Alternate Title:  “Is Dick Moffat an Old Fart?”

Tonight on Twitter, David Hager pointed me to the latest blog post from one of my favorite people, Dick Moffat.  In this post, Dick wonders whether the lack of an Office-specific conference is a bad sign.  In essence, Dick’s concern boils down to these points:

  1. There used to be an Office Developers conference, now there isn’t one
  2. SharePoint is the only Office-focused conference these days, which is a sign that the traditional Office apps are basically disappearing into SharePoint
  3. That’s a shame because not everyone wants to adopt SharePoint – Office may start to disappear from the radar of non-SharePoint shops
  4. It’s a shame in general – it reinforces the vibe that Office is no big deal anymore, after being the center of the corporate universe for so long
  5. He openly speculates throughout as to whether he is just a grumpy old fart :)

Dick, I do see a number of challenges ahead, but I also think there’s much more good news than bad news here.  Yes, I mean that even for point #5 :)

Office customers don’t view the desktop apps as a platform anymore

In the old days, it was common for even large corporate customers to build entire applications around the Office desktop apps.  For the first 5-6 years I worked at Microsoft, for instance, expense reports were done entirely in an Excel-based, client/server application.  And I think the electronic Company Store ordering application used Access under the hood.

But around 2000 or shortly thereafter, both of those solutions migrated to a server-based implementation, presented via the web.  Microsoft internal IT simply decided that was a much better way to go.

That same trend was playing out all over the world.  Desktop solutions in general were being replaced by IT departments everywhere – replaced with web-based solutions.

No one was moving away from the desktop Office applications themselves.  Excel, for instance, was just as important as ever.  But IT departments were no longer building internal solutions (like expense reporting) on top of them.

In many ways, those are the “good old days” that Dick pines for.  But they are simply never, ever, coming back.  And that has nothing to do with Microsoft giving up on the Office apps.  It’s just a lot easier to build, deploy, and maintain server-based, intranet web apps than it is to maintain an infrastructure based on the Office apps.

Good news #1:  Faster Upgrades!

The good news here is that as corporations have moved away from the desktop apps as a solution platform, it’s become easier for them to upgrade to the newest versions of Office, because the chances of them breaking a critical solution have dwindled to near zero.  They still don’t upgrade overnight, but I definitely have noticed Office being more quickly deployed than it used to be.

As an example, look how many PowerPivot folks replied and said they were running Office 2010 on Windows XP – about 30%!  You know why?  The OS is still a platform for applications, and customers are very reluctant to upgrade it at a consequence.  Office has shed that role, however, and is now “pulling ahead” of Windows in terms of upgrade cycle.  In the old days, most customers rolled out new versions of Windows and Office at the same time, so we never would have seen something like Office 2010 running on Windows XP at a 30% rate.

Good news #2:  Office 365 as a Future Platform

Let me be clear:  I am in no hurry, at all, to trade in my desktop copy of Excel for a web-based version.  I think it’s very likely that 15 years from now, Microsoft will still be selling a desktop copy of Excel, and it will be widely used.

I haven’t tried Office 365, aka Office Online, yet.  I’m pretty slow to adopt new stuff actually, both because there’s a bit of the “old fart” in me too, and because I like to let products “cook” for a version or two before I invest in them – PowerPivot is a rare exception for me.

But Office 365, in a future version, offers us the promise of Office becoming a relevant platform again.  Server-based platforms > client based platforms – the market has spoken pretty clearly on that, and I agree.  So the fact that Microsoft is investing heavily in a server-based version of the Office suite is very good news.  Let’s give them a year or two and see how things are shaping up.

Good News #3:  Does Not Require Internal SharePoint Adoption

Make no mistake, SharePoint is going like gangbusters.  Their conferences are nothing short of amazing, as I noted in one of my first few blog posts ever.

But to be an Office 365 customer, you will not have to be a SharePoint customer.

Will we someday see a year in which both Office 365 AND SharePoint each have a big conference in their honor, or perhaps a joint conference in which both get equal billing?  Yes, I believe we will, and it may not take very long.

Remember, conferences only make sense for platforms.

Good News #4:  More Frequent Feature Additions and Bug Fixes

Awhile back I was talking to some of my former colleagues in Office.  We were discussing support for a particular feature – something that is missing from the current SharePoint 2010 release of Excel Services.

Here’s the relevant part:  at one point they mentioned that it would likely be a lot easier for them to add that feature to Office 365 than it would be to work it into a service pack of SharePoint 2010.  That makes sense when you think about it:  it’s the same reason why server-based solutions are so appealing in the first place.  With server-based apps, you own every piece of the puzzle.  Bing and Google don’t need to test and deploy a new monolithic desktop suite every time they want to make a fix or improvement.  And neither will Office 365.

Challenges

It’s not all fun and games in this future of course.  First, if they want Office 365 to become a platform, they have to have a fully-featured API.  Excel Services programmability has made great strides in 2010 for instance, but it’s a LONG way away from being as complete as the client OM.

And when they get such an API ready to go, um, how is it going to relate to the desktop OM?  Like I said, I think desktop Excel is here to stay.  Designing any sort of programmability story that is sensible and coherent across desktop and server is going to be a gargantuan task.

Lastly, I wonder whether Office 365 is architected in a way that’s conducive to all of us tinkerers uploading our custom applications.  Even Windows and SQL Azure do NOT give you remote desktop access to the server machines for instance, and neither will Office 365.  They will have to provide another way to upload solutions, which should be easy.  Making sure those solutions can’t bring down the servers (that are being shared by other customers), overly tax the servers, read other people’s data, etc. – that’s not easy.

Is Dick, in fact, an old fart?

In my view there are two key indicators of being an old fart.  One is constantly complaining that the old days were so much better than today, even if there’s plenty to be optimistic about.  And yes, Dick DEFINITELY meets this criteria :)

But the other key indicator is a stubborn refusal to adapt and adopt.  When I first met Dick and heard him complaining about the declining role of desktop Office as a platform, I thought he was going to cling to that stance and go down with the ship.

Six months later, guess who had become a rabid early adopter and vocal supporter of Access Services?  Yep, Dick Moffat.  For someone who complains so bitterly about the state of things, you’d never expect him to be ahead of the curve like he is.  I suspect the same was probably true when they replaced Excel Macro with VBA in the 1990’s.  I bet Dick complained a lot about that transition while busily learning how to do acrobatic things with the Range object :)

You talk a good line Dick, but I’m onto your little game now.  I predict you will write your first line of Office 365 code before I do.  Which is good, because you and I will still be helping each other then, too :)

Dick Responds!

This is in a comment below but I thought I’d include it in the post directly so that more people see it:

Yes Rob … You nailed it !

I AM an old fart (and a second ago I proved it to myself once again :-( ) but that only means that new technologies have to prove themselves to me before I’ll jump on the bandwagon. I also believe strongly that “Traditional” Office apps (most notably Excel and Access) are not only still relevant on their own but are actually the most significant potential source of content for SharePoint going forward. If Excel and Access aren’t used with skill then the whole SharePoint thing comes crashing down.

At the same time, yes, making it so that MS only talks about Excel and Access in the context of SharePoint is great marketing but I think it’s a dis-service to the customers and potential customers of Excel and Access themselves. As you say – nit everybody is going to be going to SharePoint soon (many never) so does that mean that they are cut out of the messaging around Excel and Access? Doesn’t seem right to me somehow.

I JUST got off a middle of the night call to Europe with a client who uses an Access Services database I developed for them… they also have a SharePoint site that consumes some info from the Access Services database. They NEVER considered the idea of integrating them directly and so so alternatively they are moving data from my database to the SharePoint one manually using Excel exported data from my site into theirs. I suggested we integrate them directly so the SPsite draws data from my Access Services database directly. They never thought of that before. I am now looking into that capability for them and will be making a proposal. THAT’S the kind of messaging we need out there. Without knowledge of the real capabilities of Excel and/or Access then it’ll all be moot – and yet another opportunity will be frittered away.

And don’t get me going on the messaging around PowerPivot :-)

Dick


PowerPivot Accelerators: The Story (And Announcing: the Private Beta)

February 2, 2011

 Pivotstream's PowerPivot Accelerators

Sunday night update:  we’ve received a lot of exciting interest and have identified about 15 great beta sites so far, representing a broad cross section of the industry – BI and SQL pros, SharePoint pros, and Excel pros who are “growing up” into BI and SharePoint via PowerPivot.  Great international representation, too.  Still looking for a few more participants, so drop us an email – beta@pivotstream.com

Remove hardhat.  Pick up thinking cap.

Short version:  We have some more toys to share.  Skip to the end if you want in on the beta.

Long version:  In August 2009, while still a member of the PowerPivot team at MS, I took off my product design hat and put on my “user” hat.  From Cleveland, far removed from the internals of the product team in Redmond, I started this website, and embarked on The Great Football Project.    I was as curious as you how well the product would work.  Maybe that’s hard to believe, but I promise it is 100% true :)   
 
I think it’s fair to say that applying a platform like Excel, SharePoint, PowerPivot, or SQL tends to foster a very different kind of expertise than that fostered by building or designing it.  You become much more familiar with the gaps, in particular, which is why those product teams listen so carefully to customer feedback (much more so than the Word team needs to, for example).

So I dove in, learning things at every turn.  About six months later, armed with the knowledge that PowerPivot performed VERY well in real scenarios, I dove in even deeper:  I left MS to join Pivotstream, where we started prototyping our PowerPivot infrastructure and models even prior to the product’s final release date.

“Gentlemen, we can rebuild it.  We have the technology…”

We started finding things in PowerPivot that we wished we could change.  Not earthshattering things.  “Last mile” type things – gaps in the feature set that made a big difference in practice.

That’s when my background as a software engineer became useful again.  Imagine working your whole career on MS products, where every day, your job is to identify flaws and opportunities to fix and fulfill.  And working right next to the people who actually reshape the products every day, as if the software were made of clay – stubborn clay, but clay nonetheless.

That breeds a certain optimism, a refusal to simply accept things as they are.  Instead of saying “crap I wish they had put X in the product,” my conditioned response is “hmm…  how hard can it REALLY be to fill that gap ourselves?”

Yes, it almost always turns out to be harder than it seems up front, sometimes MUCH harder.  But without that optimism, we probably wouldn’t have gotten started.  So it serves a purpose right?  Programmers everywhere are shaking their heads in disagreement, to which I reply, “Hey, why are you reading this?  Get back to coding!”  (Kidding.  Sorta.)

The Results:  A Complete System

I’ve already shared one of the projects with you:  the trimmed-down SharePoint pages optimized for the BI portal role.  And then their subsequent beautification.  Those have been in production for us for a long time now.

That’s just the tip of the iceberg though.  There are many other things we have done to improve our efficiency, or our customers’ experience, or both.  We are very proud of the results.  Here’s a glimpse:

  1. Bulk Workbook Modification – when we started out, we would have to manually edit double-digit numbers of workbooks by hand whenever we wanted to make a change.  The same was true if we wanted to add or modify a lot of measures in bulk.  Not only was this inefficient, but also error-prone.  Today, our workbooks are very rarely touched by human hands.  We queue up a change and kick if off.
  2. End to End Data Refresh – our nightly/weekly refresh process is completely automated, from FTP delivery of new data from clients, through SSIS, PowerPivot refresh of “Core” workbooks (triggered only when the underlying SQL sources are ready!), and automatic refresh of the “Thin” workbooks connected to them.  We can schedule Cores to refresh more than once per day.  In fact we can manually trigger a “right now” refresh of Cores in bulk, with basically one click.  In the Thins, we can even increment Date slicers to the latest value :)   Status updates are automatically emailed out to our team.  And the “refresh on open” problem is a thing of the past.
  3. SharePoint Tools – there’s also a reasonable amount of drudge work involved with SharePoint stuff, particularly publishing and linking pages and workbooks.  You may not notice this, depending on how many workbooks you have, but for us, well, we have far too many to be clicking around in the SharePoint config UI all day, so we’ve built tools for that, too.

Even better:  these things will all remain valuable once SQL 11 / Denali ships, and we have things like BISM and Crescent to play with.  In fact I hear rumor that the SQL CTP2 beta release may be just around the corner…  are you pumped?  I’m pumped.

The Private Beta:  Now Taking Applications

All of that stuff is working great for us internally.  And just like with the SharePoint Pages, we’d like to share them with the community and recoup some of the development costs.  Getting these components ready for broader usage IS more work, however, and we’d like to recruit a small group of early beta testers to kick the tires.  The Accelerators won’t release until they are ready.

If you are interested, please send an email to beta@pivotstream.com and specify:

  1. Your Name
  2. Company Name
  3. Current Usage of PowerPivot – personal or organizational, prototype/exploratory or already in production, and whether you are using PowerPivot for SharePoint or not
  4. Which areas you are interested in testing (1-3 above)
  5. Website(s) – your company website and/or blog URL if applicable

We’re going to keep this first round kinda small, maybe no larger than 10-15 participants.  So get your emails in, we’re anxious to start selecting the group :)


PowerPivot Scheduled Refresh Pt 3: Thin Workbooks

February 1, 2011

 
PowerPivot Thin and Core Workbooks

What is a “Thin” Workbook?

At Pivotstream, we came up with the term “thin workbook” to describe an Excel workbook that does not contain a PowerPivot model, but that DOES connect to another workbook that does, which we call a “core” workbook.  In other words, thin workbooks use core workbooks as their data source.  Others have called this “hub and spoke,” and I may even like that phrasing better, but hey, we’ve been using “thin and core” at Pivotstream for nearly a year and it’s hard to change.

So, a thin workbook contains pivots, charts, cube formulas, etc. – it’s a report-only workbook.  And whereas our core workbooks may be 200 MB or more, our thins are often 5 MB or less.

Why create thin workbooks?

Here are a few reasons why you may want to use a thin workbook approach.

1) To avoid duplicating data.  If you want to create multiple workbooks that offer different views of the same data, which we do all the time, you probably don’t want to duplicate the PowerPivot data across all of those workbooks.  That duplication needlessly burns RAM on the server, takes extra time to run scheduled refreshes, and also increases your maintenance work (ex:  if you want to modify a measure, you have to do it multiple places instead of one).  I *highly* recommend you never duplicate PowerPivot models like that, and thin workbooks are a good alternative.

2) To lock down data.  In many ways this is another flavor of #1.  At Pivotstream we often create one workbook that contains a full set of measures, and then another that omits certain measures that are particularly sensitive.  Both can use the same core workbook as a data source, but if you properly lock down your permissions on SharePoint, you can give one set of users a different level of data than another set.

3) To avoid the refresh on open performance problem.  In the last post about refresh, I pointed out that the PowerPivot refresh service on SharePoint kinda “cheats” a little bit – it refreshes the data in the PowerPivot model, but the data in the pivots (the data in the Excel sheets themselves) is untouched.  To work around that, PowerPivot for SharePoint sets the “refresh on open” flag, forcing Excel to refresh all pivots whenever the workbook is opened.

That can be slow.  In many cases in our work at Pivotstream, it is VERY slow.  Since the PowerPivot refresh service does NOT touch the thin workbooks, “refresh on open” never gets set.

Of course, that creates another problem doesn’t it?  The data in the Excel sheets then remains stale, with no automatic way to refresh it.  That’s no good.

But we are busy little beavers at Pivotstream and we don’t give up easy :)   We started working on this problem in March 2010.  More on this later.

How do you create thin workbooks?

The first step is to publish your “Core” workbook to SharePoint.  You CANNOT connect a thin to a core sitting on your desktop – the desktop version of the PowerPivot engine does not support cross-workbook connections like that.

Once you have your Core published, there are two primary ways to create a thin workbook:  Start from Scratch, or Hack like Mad.

Boring:  Start from Scratch

In this method, all you do is go to the PowerPivot Gallery and click the highlighted button:

Creating a Thin PowerPivot Workbook from the Gallery 
…and that gets you started in Excel, in a new workbook, with a blank pivot connected to the Core model.

More Fun:  Hack Like Mad!

For those of you who prefer to color outside the lines, here’s another technique that I often find helpful.  You’ve created a single Core workbook, and it has lots of report sheets in it.  You don’t want to start from scratch and throw all of those report sheets away, right?

So here’s what you do:

  1. Create a copy of the workbook.  Just copy/paste the file in Windows Explorer.  Name one file something like Core.xlsx and the other Thin.xlsx
  2. Delete pivots from the Core workbook.  Two reasons for this.  One, it reduces confusion – when you open the Core, you know it’s the core because you don’t see any report sheets.  But deleting pivots also makes the scheduled refresh process on SharePoint run faster – after PowerPivot refreshes the model on the server, it then needs to refresh the thumbnails in the gallery.  To do that, it asks Excel Services to open the workbook, which triggers “refresh on open.”  Why pay extra time and CPU penalties on reports no one is going to see?
  3. Publish the Core workbook to SharePoint.
  4. Change the connection string in the Thin workbook.  Open it up in Excel, and in Connection Properties change the “$Embedded$” string to be the full URL to the core workbook on SharePoint.  Example:

Changing the Connection String to Point to a Published PowerPivot Workbook

When you click OK, Excel performs a full refresh against the new source.  Be patient.  Once that is done, your thin no longer connects to the PowerPivot data embedded in its own file, it now queries the server copy instead. 

Of course, now you probably want to make the file smaller and get rid of all that unused PowerPivot embedded data.  Remember where that lives?  This post explains in more detail, but here’s the image of the thin workbook renamed from XLSX to ZIP:

Embedded Data - One Reason Why PowerPivot Will Not Work With Excel 2007

Item1.data is the file you want to get rid of.  But if you delete it, you will corrupt the file.  So I just create a 0-byte Item1.data file on my desktop and copy it over the top of the file inside the ZIP.

Rename back to XLSX and you have a true “Thin” workbook with all of the original report sheets intact and functional!

Summary, and a hint at the next post

Thin workbooks are a VERY useful tool in a PowerPivot system.  For Pivotstream, they are absolutely essential.

The only trick, as mentioned above, has been getting those thin workbooks to always be up to date.  I’m happy to say we have that problem solved.

I’m pretty sure others are going to need this sort of thing, too, as well as some other internal tools and components we have built.  If you have interest in beta testing our PowerPivot Accelerators, as we are calling them, watch for the next post :)