More Live PowerPivot Web Apps!

February 9, 2012

 
Excel Pro Turned Web Developer with a SINGLE CLICK!  MUHAHAHAHA!

“IT’S ALIVE!!!!”
(Just your average Excel Pro after converting
his first PowerPivot Workbook into a Web App)

I’ve got another article about to go live on CIMA Insight, but I’m gonna jump the gun a bit and post basically the entire thing here ahead of time.

At Pivotstream we recently went live with our first full-time demo site for Hosted PowerPivot.  We’re going to be adding to it over time, but it’s got enough on it already that I think it’s worth looking at – it shows that “spreadsheets have become live web applications” thing that has to be seen, live, to really sink in.

Customizable Homepage

Hosted PowerPivot Demo - PowerPivot for SharePoint Made Simple

Pretty self-explanatory – customizable content plus a menu of applications.  Link below. 

(OK, one note:  I say “customizable” but customization is only allowed for Authors/Owners of the site.  What you are seeing here in the Consumer experience, and Consumers cannot customize this home page).

https://insights.hostedpowerpivot.com/sites/Demo/Pages/default.aspx 

App #1: Based on Microsoft’s “AdventureWorks” Sample Data Set

This is the workbook from the Budget vs. Actuals Part One and Part Two posts. Here’s a picture of that same workbook after it’s been saved to a SharePoint web server and then accessed in my browser:

Hosted PowerPivot Demo - PowerPivot for SharePoint Made Simple

Notes on this application:

  1. This is just an XLSX file, created with PowerPivot and then saved to the server
  2. But it’s rendering in my browser (Firefox in this case), and the XLSX is NOT being downloaded to my computer
  3. I do NOT have to have Excel installed on my computer in order for this browser application to function
  4. This picture is of the menu page of the app, which is just a worksheet with the gridlines and headers turned off
  5. The graphics are two image files inserted into the sheet via Insert.Picture on the ribbon
  6. The four hyperlinks are merely links to other worksheets within the workbook

If I click the “Sales vs. Budget” hyperlink, I am taken to the report I built in last month’s article:

Hosted PowerPivot Demo - PowerPivot for SharePoint Made Simple

Click the slicer – it works Smile

Try this application out here:

https://insights.hostedpowerpivot.com/sites/Demo/Pages/Adventureworks-Sample.aspx

Application #2: Retailer Overlap Competitive Analysis

This application is based on two real-world data sets. One is a list of the addresses of almost every retail food/drug store in the United States. The other is a detailed list of demographic information about every ZIP code (postal code) in the United States.

Blend them together in PowerPivot and you get an application like this:

Hosted PowerPivot Demo - PowerPivot for SharePoint Made Simple

Note that I have selected the two warmest temperature ranges, circled in orange. If I click the link at the top of the sheet I then see the following analysis:

Hosted PowerPivot Demo - PowerPivot for SharePoint Made Simple

Notes:

  1. I have the application set as if “I” were Walgreens, and evaluating my competitors’ geographic overlap with my Walgreens retail locations
  2. The first column indicates to me that CVS competes with me (Walgreens) for 41.5% of the potential customers that I try to reach
  3. In other words, 41.5% of the people who live near a Walgreens also live near a CVS
  4. The second column reports that Walgreens competes with me for 47.3% of my customers in Warm locations
  5. So the first column ignores the slicer selections I made on the menu page, and the second column respects them!
  6. The third column represents the delta between columns 1 and 2. Interesting, for instance, that Rite Aid does NOT compete with me at all really in warm locations
  7. Try it out, slice away – there are many ways to slice and analyze this data set

Application is located here:

https://insights.hostedpowerpivot.com/sites/Demo/Pages/Retailer%20Overlap%20Analysis.aspx

Application #3: CRM Analyzer

This one is also based off of a sample data set, but it is one pulled from a popular CRM package and therefore represents real-world value.

Hosted PowerPivot Demo - PowerPivot for SharePoint Made Simple

Note that there are multiple reports in this application as well, plus a menu page, but I’m just showing one here for space reasons.

Application can be tried out here:

https://insights.hostedpowerpivot.com/sites/Demo/Pages/CRM-Analyzer.aspx

Application #4: UFO Sightings

Many of you have seen this one already, on Mr. Excel’s Hosted PowerPivot site, but we put it on this demo site as well.

Hosted PowerPivot Demo - PowerPivot for SharePoint Made Simple

Application is located here:

https://insights.hostedpowerpivot.com/sites/Demo/Pages/UFO-Sightings-Real-Data,-Humorous-Conclusions.aspx


In the Browser, Aesthetics Yield a Greater Return

January 10, 2012

 

A PowerPivot Doc in the Browser is an Application   
A Spreadsheet in Excel Services Is No Longer a Document,
It’s an Application

I thought today was going to be a “handful of mini posts” kind of day but then this post blossomed into a bit more than I thought it would, which is a good thing.

Normal Spreadsheets are Usually Ugly and That’s OK

It’s true:  most Excel pros are not artists.  I certainly am not.  We’re number folks first and foremost, and our jobs haven’t historically placed top-level emphasis on aesthetics either.  So we don’t spend much time on it, typically.

Besides, Excel itself isn’t pretty.  Back in the 80’s or 90’s, even if you made a spreadsheet look fantastic, well, it was still loaded in Excel.  So you got all those lovely “battleship grey” toolbars, the title bar, etc.:

Spreadsheets of Yore Were Also Made Ugly Just by the Excel Frame

Spreadsheets of Yore Were Also Made Ugly
Just by the Excel Frame Itself

I want to be clear:  This is NOT a post that is going to encourage you to run out and start putting lipstick on all of your spreadsheets. 

Instead, I am going to make the case for why the game changes significantly (for the better too) when you switch to using a server (or a cloud hosting service like ours) to share your work.

The “Excel Frame” Has an Enormous, Underappreciated Impact

Hey, you might say that the old-style Excel screenshot above is an unfair example, since Excel 2007 and 2010 replaced menus and toolbars with the Ribbon.  But in a crucial way the Ribbon is NO different really – the point is that either way (ribbon or menu/toolbars) the Excel frame is NOT YOURS.  It belongs to Excel.  And no matter how much work goes into the document itself, the user of that document still thinks you made them a document.

Let that sink in for a minute.  Because the user of your spreadsheet thinks they are “using Excel” and not “using an application built by my favorite Excel pro,” you are receiving a hidden benefit AND a hidden penalty:

  1. There are many things you will never be blamed for as long as the consumer thinks of Excel as the application.  Hey, the overall experience just feels kinda clunky.  “No big deal, that’s just how Excel docs always are.”
  2. But you also don’t get nearly enough credit, psychological impact, or perceived importance that your work deserves.  Seriously, your work drives your organizations.  If everyone thought of you as a programmer (which you are, even if you don’t write macros), you’d be viewed differently.  But people who produce documents are often viewed as “Excel Monkeys.”  Honestly I think Excel pros are, for the most part, underpaid relative to their true importance.

When you switch from mailing spreadsheets around to publishing spreadsheets on SharePoint, well, both of those go out the window.  Well, if you do it right, anyway.

SharePoint Brings Its Own Frames!  Ack!

OK, so you switch over to using SharePoint as your publishing mechanism.  Does that get rid of the Excel frame?  Well not really.  It just gives you a new Excel frame in your browser:

See all of those highlighted elements?  Just another Excel frame, ported to the browser.  Complete with File tab, toolbar, the .XLSX extension blazoned across the top, and even a warning bar.

Not to be left out, SharePoint adds some of its own at times:

So, you gotta get rid of those.  And that means customizing SharePoint.  If you’re a SharePoint pro that’s mostly not too difficult, but even then it likely will take you some time to get it tuned just right (we’ve been making tweaks now for two years).  And if you’re not a SharePoint pro, well, you are going to need one.

(For more info on the details of these SharePoint elements and what we’ve done to modify/suppress them, see this post and this post).

Going Frameless Turns “Document” Into “Application”

In our Hosted PowerPivot offering, we’ve got all of that suppressed, and the only “frame” you see is just the browser and a typical web header.  An example:

PowerPivot Application Built by Pivotstream

The Consumers of This Application Neither Know NOR Care that it Was Built in Excel

The idea for this post struck me yesterday as I was putting together this sample workbook (based on Microsoft’s AdventureWorks data set) that we are going to start including in all of our HostedPowerPivot sites as a tutorial:

Sample Hosted PowerPivot Workbook

Sample Workbook v1 For Hosted PowerPivot

Or try this humorous example (based on real data) that examines UFO sightings – click image to view the application on Mr. Excel’s HostedPowerPivot site.

Live PowerPivot Application on HostedPowerPivot

Click Image to View the Live APPLICATION

Completing the Illusion:  A Few Simple Steps

Once you’ve gone frameless, there are a few simple things you can do to complete the transformation from document to application.  Neglect these and your “frameless” would-be application screams “spreadsheet” to the audience.  Follow them, and even if you’re not terribly artistic, your work will be perceived very differently:

  1. Turn off gridlines and headers.  It’s not hard.  Two checkboxes on the View tab of the ribbon, but do that for EVERY sheet the consumer sees.  Every single one.  Crucial.
  2. Hide or delete all sheets you don’t want them to consume.  Don’t leave extra blank tabs in there named “Sheet3” OK?
  3. Insert some images.  So important!  Your company logo.  Your client’s logo.  Something.  And make sure you use the Insert ribbon to do this!  Simple copy/pasting an image into a worksheet, in my experience, seems to result in that image NOT showing up in the browser.
  4. Line up slicers, charts, and tables.  Takes just a minute or two to make sure the top of your slicers are even with the top of your chart, etc.
  5. Don’t neglect number formatting.  If something is a currency, format it as a currency.
  6. Use conditional formatting.  No need to overdo it, but conditional formatting turns a boring black and white grid (a pivot) into an inviting surface that is actually fun to look at.  Plus, trends, patterns, and outliers jump off the page much more readily.  I’m especially fond of data bars, color scales, icon sets, and when I have the time, sparklines.
  7. Create a Menu sheet (Table of Contents), and use hyperlinks for navigation.  Yes, the sheet tabs are visible.  But why force people to use them?  There are a million reasons why sheet tabs are disproportionately old fashioned and cognitively difficult.  When it comes to navigating around a web application,  nothing comes close to a hyperlink. 

“I’ll Take ‘Hyperlinks Between Sheets’ for the Win”

Let’s focus on that last one.  Did you know that you can hyperlink between sheets in a workbook?  I worked on Excel for years and never really realized this.  Our CEO at Pivotstream pointed that out to me, and it works on the server too.

Hyperlinking Between Sheets in Excel

Hyperlinking Between Sheets in Excel

This lets you create menu sheet like the AdventureWorks sheet above, as well as this one at the beginning of the post:

This Is Actually a PowerPivot Menu Sheet aka Table of Contents

This Is Actually a PowerPivot Menu Sheet (aka Table of Contents)

Cool huh?  Those chart thumbnails are IMAGES.  The hyperlinks above them take you to the full-page interactive sheets that host each of those chart views.

Formatting Macros

I have a number of macros that help me do some of this stuff, and in an upcoming post I will share some of them, once I have time to organize them a bit.

In the meantime, here’s a real simple one whose intent should be obvious:

Sub HideGridAndHeadersOnAllSheets()

    Dim oSheet As Worksheet
   
    For Each oSheet In ActiveWorkbook.Worksheets
        If oSheet.Visible = xlSheetVisible Then
            oSheet.Activate
            ActiveWindow.DisplayGridlines = False
            ActiveWindow.DisplayHeadings = False
        End If
    Next

End Sub

How many of you use macros, by the way?  I’m really curious.  Drop me a comment if you would and just say yes/no.  I will also put up a survey at some point if I get industrious.

Conclusion

I know that not everyone who has embraced PowerPivot for Excel has started using PowerPivot for SharePoint yet.  That’s changing, but it takes time.

For those of you who are starting that transition, I’m very excited for you.  I know it sounds weird.  But you cannot appreciate how much more impactful your work “feels” as a web application until you see it in action.  And this post is aimed at helping you reap that benefit.

For those of you who are yet to start down that path, file this one in the back of your mind for later.


Try Out Sort by Slicers, UFO’s, in Browser, No Login!

October 25, 2011

 
UFO Tracking in PowerPivot

Explore UFO Data.  And Try out “Sort by Slicers” for Yourself.

OK folks, Halloween is in one week.  Keeping in the spirit of Pivotstream “honoring” holidays with themed workbooks on the web, I think it’s only fair that we post the UFO workbook on Mr. Excel’s Hosted PowerPivot site.  So we have.

It’s a 26 MB workbook, but no worries – you don’t have to download it.  Just interact with it in the browser like it’s a web application…  which it is.  Click around, explore it, see if you find something that cracks the secret of Area 51.

And yes, this is the same workbook that I used for the Sort by Slicers post, the follow-on Sort by Alphabetical post, and the Hallucinogens and UFO Sightings post.  But we’ve added a few more report sheets since then too…

NoLoginRequiredNo Login Required!

Oh, I can hear the whining already.  “Oh, I gotta go look up the password you posted a few weeks back.  That’s hard.  I have more important things to do today.” 

Too much work for you eh?

Well I’ve got a deal for you.  We can now do purely “anonymous” demo sites.  That’s right, no login required folks.  Not even if you’re Phil himself.

Just click this link, and you’re in!

At the moment, I believe this to be the only no-login PowerPivot internet site on the planet.  Well, almost.  We’re running a couple of other no-login demo sites for some of our Hosted PowerPivot customers, but they’re not ready for me to publicize that.  And no, you can’t find them either, because they are NOT using Pivotstream URL’s – they are using their own URL’s even though they are using our servers.

They will clear me to share those URL’s soon though I hope Smile

Not Enough for You?  OK, Download the Workbook and Modify It!

There have been some requests for me to share the UFO/Sort-by-Slicers workbook.  Well, you can’t download it from Mr. Excel’s site.  That’s one of the benefits of PowerPivot for SharePoint – you can publish workbooks and let people interact with them, but DENY them the ability to download the whole workbook.

That’s pretty important in most business scenarios.  There’s often a LOT of sensitive data in a PowerPivot workbook, and very often, even the formulas themselves represent valuable intellectual property – both are certainly true in Pivotstream’s workbooks for Walgreens-Duane Reade, for instance.  You can download a snapshot, which is almost like “printscreen to Excel,” but you can’t grab the good stuff.

So you’re gonna have to get the PowerPivot workbook itself somewhere else.

OK, I put the downloadable workbook here.

Go get it, have fun.  I would not call it our most elegant work – this was for fun even though it came from a real data set.  So there’s a lot of room for improvement.  Which brings me to…

Share Your Improvements, and I’ll Post the Best to the Site

And since I shared the workbook with you, well, please share your modifications with me!

Send screenshots to info@pivotstream.com, or send links to locations like dropbox – I don’t think our email will accept 26 MB files. 

I’ll take a look.  If I see any really cool mods, Mr. Excel and I will post them to his site for the world to see Smile

ShareYourself

“Share Yourself Completely”


Mr. Excel Makes His HostedPowerPivot Site Public

September 22, 2011

 
image

PowerPivot Workbooks as Interactive Web Apps – Try it Yourself!

Bill Jelen and I talk a lot about a few recurring topics.  One of our favorites:  it’s one thing to be told that PowerPivot workbooks can be transformed into web applications with a simple Save As, and quite another thing to see it for yourself.

I don’t know why there’s a difference, but there is – you’re looking at something in your browser, it’s an interactive application, and then you remind yourself, “oh yeah, I built this in Excel,” and a light bulb comes on.  I see that effect every day when I show people for the first time.

Try Out Bill’s Apps, and Send in Your Own

To test that on a broader scale, Bill decided to take his site on HostedPowerPivot.com and open it up to the community, which I think is really quite cool.

Right now he only has two applications shared out publicly – he’s going to let YOU send in workbooks that he will review and publish.  He’s also got some PowerPivot gear to give away.

Watch the short video here for a quick tour, explanation, and instructions on submitting workbooks

Then, go check out the site itself:

http://mrexcel.hostedpowerpivot.com

Username:  MRXL@pivotstream.com
Password:  P0werP!vot

NOTE:  that first “0” in the password is a zero, not a letter.

Have fun!


Couldn’t Resist… Parameterized PowerPivot Report!

July 20, 2011

In the intro to David’s guest post below I mentioned some new applications we are building.  Check out a PowerPivot report where, in the browser, you can type in a SKU number and get a full dashboard for just that product!

Parameterized PowerPivot Report in SharePoint

Click for Larger Version

Much respect to our resident Data Junkie Monkey, aka DJ Monkey, for pulling this together.


Three quick news updates

May 27, 2011

A few quick items for this Friday, then I’m off to Cedar Point for some much-needed coaster thrills:

1) HostedPowerPivot Interest – Wow!

We’re blown away by the response to HostedPowerPivot.com.  I’ve spent multiple hours on the phone today with longtime blog readers – an added benefit of this is getting to know some of you better.  Very cool, seeing all the different usage cases people have in mind.

2) Webinar with Rackspace June 8th

I’m going to be doing a PowerPivot webinar with Jeff DeVerter, SharePoint guru at Rackspace, on June 8th.  It’s going to be more of an intro to PowerPivot, so longtime readers here may find it elementary.  But there will be some fun demos, and Jeff is a really dynamic personality, so I am looking forward to it.

Click here to view the agenda, and to reserve a spot

3) Part two of “portable formulas” is live on the Excel blog

I’m 100% serious that formula portability is one of the top 3 benefits of PowerPivot over normal Excel, it’s just taken me awhile to figure out how to explain it.

Click here for part two

…and here if you missed part one


HostedPowerPivot.com from Pivotstream & Rackspace

May 25, 2011

Click for HostedPowerPivot.com
No, we're not villains.  But the quote was too perfect to pass up!

“At last we will reveal ourselves to the Jedi Bee-I.  At last we will have shared intelligence.”

A secret long kept, finally revealed

It’s a recurring theme – I see it in my training/consulting practice, in my inbox, in the survey results, and at events: 

“We LOVE PowerPivot.  It’s a perfect fit for our analysis and reporting needs.  But our company has not yet adopted SharePoint, and we don’t have the in-house expertise to stand up and support PowerPivot for SharePoint.    We just want the simple beauty of PowerPivot, we want it now, and it’s frustrating that we can’t have the full system yet.”

I hear you.  That is precisely where we found ourselves when I joined Pivotstream.  The lack of a “turnkey” solution to that problem meant we had to go build it ourselves.  And our core business has been running on our internet-based PowerPivot infrastructure since last summer.

But now, one year and two hosting providers later, we are finally able to share what we’ve built with the community.

A Long, Long Time Ago, In a Conference Room Far, Far Away…

OK, it was February, in San Antonio.  John Casey and I were at Rackspace headquarters for two full days to pitch an idea:  that an all-in-one, customized-to-your-needs, zero-hassle PowerPivot for SharePoint infrastructure would be a very valuable thing to the world at large.

We’d chosen Rackspace based partly on their reputation for support, but primarily because they had the most SharePoint expertise in the hosting business.  SharePoint, after all, is probably the most complicated part of running a PowerPivot server farm.  We were already moving our own server farm over to Rackspace at that point, but now we were pitching them on a partnership.

It’s a dicey proposition, walking into someone else’s offices knowing that you have to start from scratch.  We planned to cover the dynamics of the BI market, past and present, Excel’s place in it, Microsoft’s first-ever total alignment on a strategy, and why imagePowerPivot was going to change the world.  That’s a tall order for anyone to digest or believe in a short two days, no matter how fervently I believed in the message myself.

I underestimated them.  They understood perfectly.  We ended up meeting with 10-12 members of their leadership team over those two days, transitioning from “here’s a cool idea” to “here’s how we can execute.”

Keeping this a secret has been the longest three months of my life.  I am stoked that the waiting is over.

Want the short version?

Being that this is a blog – my blog, specifically – and that I love telling stories, my aim here is to describe how we got here – motivations, steps along the way, etc.

But if you just want to get to the “meat” of this, and/or request more information, go ahead and visit HostedPowerPivot.com:

Click for HostedPowerPivot.com

Ok, back to the story.

Thanking Rackspace

I’m pretty sure we could not have done this HostedPowerPivot thing with anyone else, although I did not fully understand that going in.

We’ve been running our core PowerPivot platform in a Rackspace data center for four months, and the level of support we get from them is night and day different from what we had in our last data center.  They advertise “fanatical support,” and I’m a believer now.

As fantastic as that is, though, the word “support” doesn’t capture what continually impresses me.  I keep coming back to the human element – the real people on the other end who are acting like human beings and not cogs in a machine.  I’m not accustomed to big established companies, especially infrastructure companies, maintaining a nimble, entrepreneurial vibe, but that’s what I’ve found here.

For instance, does this sound like “support” to you?

Me:  “Hey Rackspace, we’ve found some unexpected PowerPivot performance results on this hardware set.  We’re now running some tests on every hardware platform we’ve got.”
Rackspace:  “Are there some other hardware options we can try out for you?  We’ve got access to a bunch of stuff here you know.”
Me:  “YES.  You’d need to install PowerPivot and run a bunch of tests on each machine, do you have time for that?”
Rackspace:  “No problem.  Send us the instructions and we’ll try it on 10 different machine types.”

Rackspace:  “OK, here’s a detailed spreadsheet of our results.  Three test runs for each unique config, reported separately and then averaged.”
Me:  “Did you say spreadsheet?  I think I’m in love.  We’ll correlate that with our other results.”

Me:  “OK based on all results, the best query performance would be achieved on a non-standard config, one with the following properties…  is that machine something that can be built out in your datacenters as THE standard PowerPivot server?”
Rackspace:  “Hmmm…  we’ll look into it and get back to you.”

Rackspace:  “Yes, we have approval to build that out.  Should we order one up for testing purposes?  We’ll have to have some new equipment delivered from the hardware vendor, might take a few days.”
Me:  “Yes please.”

Rackspace:  “Test machine racked and running.  And uh, I think you will be pleased.  It’s blowing the socks off of everything else we tested!”
Me:  “I love it when a plan comes together.  Gentlemen, we have ourselves a PowerPivot server.”

Thanks guys.  Too many of you to name specifically, but you know who you are Smile

Step 1:  Register Domain.  Step 2:  Submerge in PowerPivot for a year

True story:  Jeff Elderton, our CEO at Pivotstream, registered the domain HostedPowerPivot.com before we even decided I was going to sign on.  It’s been in our plans from the beginning.  But before we could credibly do such a thing, we first had to apply the technology ourselves, for our own core business.  We dug into that while PowerPivot was still in beta, as our sole focus.

Along the way we had to solve all of the common problems everyone will hit.  We’ve written software to plug the gaps and provide a professional aesthetic.  We know how to “capacity plan” specifically for PowerPivot.  We’ve even figured out that certain hardware configurations can dramatically outperform the most commonly-used server configs.  We learned a lot more than we expected to.

All of that was expensive and time-consuming of course.  But it was absolutely worth it.  The things we deliver to our customers simply were not possible before PowerPivot.

Today, I’m pretty sure no one in the world runs a PowerPivot infrastructure of the depth and breadth of what we run at Pivotstream.  Our entire core business (subscription analytics for dozens of clients) runs on our PowerPivot infrastructure.  There’s no substitute for just doing something – I learned much more about PowerPivot from the outside, as an adopter, than I did as an insider, working on the team at MS.  That was surprising, although it makes sense in hindsight.

I’m really happy to see it all come full circle.  At his core, Amir Netz describes himself as an inventor.  I like that, I think it fits him quite well.  I’m similar in some ways, but it’s not like I will ever come up with something like the VertiPaq engine, so “Inventor” would be an overly generous description of me.  I like to think of myself as a creator.  I love creating useful things.  I love filling voids.  And this one has had me jazzed for a very long time.

1997:  Alabama 20, Vanderbilt 0

There’s one more story I’d like to tell, and it’s a bit of a cliffhanger because it deserves its own post.  Like so many other things around this blog, it all comes back to football:  there’s a connection between us hooking up with Rackspace, and the 1997 Alabama routine thrashing of Vanderbilt.

Just one of those fun little wrinkles in life.
 

Click here for THRILLING highlights. I wonder who posted these? Hmmm…

PowerPivot Accelerators: The Story (And Announcing: the Private Beta)

February 2, 2011

 Pivotstream's PowerPivot Accelerators

Sunday night update:  we’ve received a lot of exciting interest and have identified about 15 great beta sites so far, representing a broad cross section of the industry – BI and SQL pros, SharePoint pros, and Excel pros who are “growing up” into BI and SharePoint via PowerPivot.  Great international representation, too.  Still looking for a few more participants, so drop us an email – beta@pivotstream.com

Remove hardhat.  Pick up thinking cap.

Short version:  We have some more toys to share.  Skip to the end if you want in on the beta.

Long version:  In August 2009, while still a member of the PowerPivot team at MS, I took off my product design hat and put on my “user” hat.  From Cleveland, far removed from the internals of the product team in Redmond, I started this website, and embarked on The Great Football Project.    I was as curious as you how well the product would work.  Maybe that’s hard to believe, but I promise it is 100% true :)   
 
I think it’s fair to say that applying a platform like Excel, SharePoint, PowerPivot, or SQL tends to foster a very different kind of expertise than that fostered by building or designing it.  You become much more familiar with the gaps, in particular, which is why those product teams listen so carefully to customer feedback (much more so than the Word team needs to, for example).

So I dove in, learning things at every turn.  About six months later, armed with the knowledge that PowerPivot performed VERY well in real scenarios, I dove in even deeper:  I left MS to join Pivotstream, where we started prototyping our PowerPivot infrastructure and models even prior to the product’s final release date.

“Gentlemen, we can rebuild it.  We have the technology…”

We started finding things in PowerPivot that we wished we could change.  Not earthshattering things.  “Last mile” type things – gaps in the feature set that made a big difference in practice.

That’s when my background as a software engineer became useful again.  Imagine working your whole career on MS products, where every day, your job is to identify flaws and opportunities to fix and fulfill.  And working right next to the people who actually reshape the products every day, as if the software were made of clay – stubborn clay, but clay nonetheless.

That breeds a certain optimism, a refusal to simply accept things as they are.  Instead of saying “crap I wish they had put X in the product,” my conditioned response is “hmm…  how hard can it REALLY be to fill that gap ourselves?”

Yes, it almost always turns out to be harder than it seems up front, sometimes MUCH harder.  But without that optimism, we probably wouldn’t have gotten started.  So it serves a purpose right?  Programmers everywhere are shaking their heads in disagreement, to which I reply, “Hey, why are you reading this?  Get back to coding!”  (Kidding.  Sorta.)

The Results:  A Complete System

I’ve already shared one of the projects with you:  the trimmed-down SharePoint pages optimized for the BI portal role.  And then their subsequent beautification.  Those have been in production for us for a long time now.

That’s just the tip of the iceberg though.  There are many other things we have done to improve our efficiency, or our customers’ experience, or both.  We are very proud of the results.  Here’s a glimpse:

  1. Bulk Workbook Modification – when we started out, we would have to manually edit double-digit numbers of workbooks by hand whenever we wanted to make a change.  The same was true if we wanted to add or modify a lot of measures in bulk.  Not only was this inefficient, but also error-prone.  Today, our workbooks are very rarely touched by human hands.  We queue up a change and kick if off.
  2. End to End Data Refresh – our nightly/weekly refresh process is completely automated, from FTP delivery of new data from clients, through SSIS, PowerPivot refresh of “Core” workbooks (triggered only when the underlying SQL sources are ready!), and automatic refresh of the “Thin” workbooks connected to them.  We can schedule Cores to refresh more than once per day.  In fact we can manually trigger a “right now” refresh of Cores in bulk, with basically one click.  In the Thins, we can even increment Date slicers to the latest value :)   Status updates are automatically emailed out to our team.  And the “refresh on open” problem is a thing of the past.
  3. SharePoint Tools – there’s also a reasonable amount of drudge work involved with SharePoint stuff, particularly publishing and linking pages and workbooks.  You may not notice this, depending on how many workbooks you have, but for us, well, we have far too many to be clicking around in the SharePoint config UI all day, so we’ve built tools for that, too.

Even better:  these things will all remain valuable once SQL 11 / Denali ships, and we have things like BISM and Crescent to play with.  In fact I hear rumor that the SQL CTP2 beta release may be just around the corner…  are you pumped?  I’m pumped.

The Private Beta:  Now Taking Applications

All of that stuff is working great for us internally.  And just like with the SharePoint Pages, we’d like to share them with the community and recoup some of the development costs.  Getting these components ready for broader usage IS more work, however, and we’d like to recruit a small group of early beta testers to kick the tires.  The Accelerators won’t release until they are ready.

If you are interested, please send an email to beta@pivotstream.com and specify:

  1. Your Name
  2. Company Name
  3. Current Usage of PowerPivot – personal or organizational, prototype/exploratory or already in production, and whether you are using PowerPivot for SharePoint or not
  4. Which areas you are interested in testing (1-3 above)
  5. Website(s) – your company website and/or blog URL if applicable

We’re going to keep this first round kinda small, maybe no larger than 10-15 participants.  So get your emails in, we’re anxious to start selecting the group :)


Hardware/Capacity Planning: From the Trenches

November 30, 2010

So How Many Servers Do I Need in My PowerPivot SharePoint Farm?

As people increasingly move up from just dabbling with the addin, and decide to start leveraging the publish/schedule/share/secure benefits of the PowerPivot for SharePoint infrastructure, I am getting this question more frequently.

How many servers?  How much RAM?  And less often but just as important…  how many CPU’s?

Time for an old joke:  “Ever hear the one about the statistician who drowned in a river that was, on average, only 3 inches deep?”

That’s the whole joke.  And as time goes on, I only find it funnier.  And funnier.  But first, some basics.

The Three Primary Server Roles

For the vast majority of PowerPivot deployments, you will mostly need to concern yourself with three server types:  Excel Services, PowerPivot Engine, and Data Source DB Servers:

PowerPivot Server Roles Summary For RAM and CPU Planning 
(Click for Larger Version)

A few notes on the diagram up front:

  1. I recommend viewing the larger version – there is information on the diagram (in the notes) that I will not repeat below.
  2. Don’t read too much into the 3/4/2 ratio of Excel Services/PowerPivot/SQL – I included different numbers of servers in each tier intentionally, to illustrate that you can scale each tier out at independently.  But, for instance, I do NOT expect you will need 3 Excel Services boxes for every PowerPivot box (you will likely need less).
  3. Although separated above, server roles CAN be combined onto single servers – for instance, many folks can get by with an “all in one” server where everything above lives on a single box.  Also, even in a multi-server farm, one of the MS-recommended configs is to combine Excel Services and PowerPivot onto a single box, and then deploy as many of those combined boxes as needed.
  4. There are some elements missing from the diagram – for instance, there is a Web Front End (WFE) role, and a SharePoint Configuration Database role, but in my experience, PowerPivot does not put unique strain on those elements (except for config db disk space).  Your mileage may vary of course.

Early Planning Efforts

Back in the Spring, I set down to the very serious task of “how much hardware will we need on our SharePoint servers in order to handle user loads?”  And naturally, I made a very serious spreadsheet to model it out:

PowerPivot SharePoint Hardware Planning Spreadsheet v1

Wow, what a spreadsheet.  I mean, I even used the =POISSON.DIST() & =BINOM.DIST() functions – clearly, such spreadsheet horsepower indicates accurate results!

Turns out, that spreadsheet was nothing more than an amusing theoretical exercise with little bearing on the real world.

There were multiple problems with that spreadsheet:

  1. It relied on outright guesses as to how often/how intensely consumers would utilize the published reports.
  2. I was not yet aware of the CPU-gobbling power of slicers, a power that makes real-world queries many times more CPU-intensive.
  3. Most importantly, it assumed peak usage would occur in the morning, when consumers are most in need of fresh information.

Solution for 1) and 2)

These are in some sense the simplest to address.  Quite simply, put up a pilot solution and observe the usage characteristics.  You will learn a lot about usage patterns as well as what that does to CPU and RAM.

But even better:  you may choose to simply ignore these factors for now and focus on problem 3 instead.  Here is why:

Peak Load is Probably Experienced During Scheduled Refresh

If you’ve looked at the larger version of the diagram above, you have seen a hint as to what I’m about to say here: 

At Pivotstream, our nightly refresh process puts far more strain on our servers than our users do, and is what we now plan our hardware around.

I will explain further, so you can evaluate whether your situation will be similar.  We receive new data on a nightly basis (typically late at night).  That data gets ingested into SQL server, and then our PowerPivot refresh process begins.

Once PowerPivot refresh begins, we have about 6 hours, tops, to get all models and reports refreshed, so that when business opens in the morning, everyone has access to fresh insights.  6 hours sounds like a lot…  until you discover that it isn’t.

A lot happens during scheduled refresh!

Keep in mind that the PowerPivot model (the embedded database living inside your PowerPivot workbooks) must first be refreshed – let’s call this phase “Model Refresh.”  This can put a lot of strain on your SQL servers just in terms of supplying the data, since PowerPivot v1 pulls a fresh copy of all tables (no incremental refresh).

As the data flows into the PowerPivot servers, a LOT of CPU power goes into compressing that data into the storage format.  During the refresh process, RAM usage steadily climbs as well.

As model refresh nears completion, RAM usage spikes upward quite a bit, sometimes by as much as 50-100% the size of the resulting workbook.  CPU usage also spikes during this phase.

Once the model is done refreshing, PowerPivot for SharePoint then triggers a refresh (via Excel Services) of the Pivots and Cube Formulas in the workbook, so that when thumbnail screenshots are “snapshotted” for the report gallery, those reflect the latest data (let’s call this phase “Pivot Refresh”). 

Yes, that process initiates via Excel Services, but that refresh immediately results in a meaty set of queries sent back to the PowerPivot servers for processing.  So, CPU usage spikes again.  And the model in question is held in RAM while that happens, so that RAM can’t be recycled into the pool to be used for model refresh.

Remember, this all happens for a single workbook.  And if you hang a bunch of report-only workbooks off of a single “hub” model workbook (as we do all the time), the amount of time it takes to finish Pivot Refresh can actually exceed the time for Model Refresh.

If you have to cram all of this into a narrow nightly window, chances are that you will need more hardware to pull it off than you ever will need for normal daily usage!

Rough Guidelines for Your Own Situation

This post is running a bit long, so I will try to be succinct in closing:

  1. If a significant percentage of your PowerPivot models and reports will be refreshed nightly, refresh is likely going to be a larger peak strain than normal interactive usage
  2. Even if you only refresh weekly (or less often), if the execution window between “data is ready” and “reports must be ready” is basically still a single night, then refresh is still likely going to drive your peak hardware need.
  3. If you think “Peak Load = Refresh Process” is likely the case for your organization, I suggest ignoring interactive usage projections during your hardware planning process, and instead developing some prototype workbooks (with rich data sets and mutliple report sheets with 100% realistic slicer sets, measure complexity, etc.), and then putting those workbooks through the refresh process to get a baseline.

Illustrated guide to Excel Services Chart Rendering

October 4, 2010

 
COMMON QUESTION:
  “Hey, will this chart render unchanged in Excel Services, or will it look different?”

MY STOCK ANSWER:  “Hmm, I forget, let’s try it out”

I hate giving that answer.  So today, I ran a little test.  Simple workbook, 12 charts that are all flavors of fancy.  Not an exhaustive test, but a reasonable sample.  Here are the 12 charts in Excel client (desktop).  Click for a larger version.

Client2 
12 Charts in Desktop Excel

Green Circle = Renders 100% identical in Excel Services
Red X = Renders less than 100% identical (sometimes small differences, sometimes large)

Now, here are the same 12 charts rendered in Excel Services:

ECS
Same 12 Charts, As Displayed on SharePoint (Excel Services)

Again, note that being flagged with an X isn’t the kiss of death – I just didn’t want to bog down in “grading” the differences.  Sometimes the changes are big (top left chart) and sometimes the changes are astoundingly small (bottom middle chart).

Either way, having an illustrated reference is going to be helpful for us, so I thought I’d share it.  Feel free to submit questions/comments/other examples.  I may expand this reference over time.