Pivotstream has a new logo

May 28, 2010

We thought it was time to freshen the image a bit.  Whatcha think?

 

pivotstream logo compact


Up for air… briefly

May 27, 2010

 
Mini-Posters-Elvis---Vegas-73722

“A little less blogger nation, a little more action please.”
-The King

Folks, I’ve been up to my eyeballs in PowerPivot these days.  Which ironically has resulted in less writing about PowerPivot.

Quite simply, the release of PowerPivot v1 to MSDN was like a huge starting gun going off.  Suddenly, all that theory became “ok, let’s put it into practice…  on 10 different fronts.”

I’ve been on so many exciting phone calls and web meetings lately.  Lots of very cool customer and partner opportunities.

Oh, and tons of “real” work, too.  Which has left less time for blogging.

But I do love the nifty community we’ve got going here and will never neglect it for long stretches.  So, while I burn the candle at all three ends, I thought I’d share some quick observations and experiences, rapid-fire style:

  1. Microblogging has its place – when I don’t have time for a full blog post, I tend to drop something into my Twitter account.  If you’re not a Twitter person, you can see what I mean here.  And if you are a “tweeter",” well, you can click there, too :)
  2. I fit 200 Million rows, a 63 GB SQL database, into a 1.3 GB PowerPivot Workbook.  Yeah, no kidding.  50x compression.  And some of the queries against the resulting workbook ran in less than 3 seconds.
  3. Another db with 120 Million rows exceeded the 2 GB file size limit – different data set.  See?  There really is no set rule of thumb for how much compression to expect.
  4. Removing irrelevant columns can dramatically shrink your file – this is well documented elsewhere, but removing columns from your data before importing them can have a HUGE impact on file size.  It can be surprisingly non-linear, too – remove one column out of ten and file shrinks by 50%.
  5. Seems like deleting columns after the fact does NOT shrink the file by as much – I have not verified this scientifically.  But I *seem* to get better file size savings by never importing columns in the first place, as opposed to deleting them post-import.
  6. Early response to PivotStream’s new offering on the PowerPivot platform has been VERY positive – yesterday we demo’d a new set of interactive dashboards, built on PowerPivot, to an existing customer who currently is served by one of our other technologies.  It got a big thumbs up, we are moving ahead with a full conversion.  And another customer is likely just a few days away from making a switch from their existing on-premise solution, too.
  7. In general, MS might not need viral adoption – when we were building the product we often talked about how bottom-up adoption by Excel users was crucial, that top-down adoption would be slow.  So far, I am seeing zero indication of that.  Everywhere I go, everywhere I demo PowerPivot, the organization I’m talking to does a collective “yes, please.”  Maybe MS could have charged for the client addin after all.  Shhh, don’t tell them.
  8. I’m seeing more willingness than in the past to embrace the new Office wave – normally, the new wave of Office products takes years to seep into corporations.  That will still be true in a lot of places.  But about 2/3 of the places I go, I find eager willingness to roll out the new products, even if only on a limited, departmental basis, in order to get PowerPivot.
  9. Any serious work on PowerPivot measures requires Notepad++ – seriously, I don’t know how I ever got by without it.
  10. Running a PowerPivot server farm is hard work – I have newfound respect for IT pros everywhere.  Don’t go it alone, folks.  We’ve brought some heavyweight talents on board to help out with SharePoint, farm architecture, and Integration Services.  I can’t imagine where we’d be without them.

-rob out


I Risk Serious Bodily Injury vs. the Dutch Menace

May 17, 2010

 
Dutch PowerPivot Menace

“My name is kasper de Jonge. I’m a BI specialist from the Netherlands. I play hoops all my life and I never lose.  Soon I play PowerPivotPro, and the world will see his defeat.”

Yeah folks, you’ve probably all seen Kasper’s guest posts.  But have you heard his voice?  As I’ve said many times, he doesn’t exactly sound like a computer nerd – he sounds more like the creation of some Soviet lab, leftover from the Cold War.

And he plays basketball. 

That’s right.  It’s not enough for Kasper to jump right into PowerPivot and subvert that beautiful American creation for nefarious Continental purposes (ok, actually, the PowerPivot team is at least 50% international in its makeup, but please, let’s not let facts get in the way of trash talk). 

No, he also has to try to take our basketball from us, too.

Well, I can’t let him do that.  Someone has to take a stand.

I recently got word that Kasper will be attending TechEd / MS BI Conference in New Orleans next month.  I issued a challenge.  He accepted.  A one on one basketball game to set the world on its proper course.  Our representatives are busily negotiating the rules.

Kasper plays hoops all the time.  Coaches a women’s team.  Watches NBA games at odd hours via what can only be black market means.  He is the modern basketball equivalent of Ivan Drago.  He is a machine.  He’s about 2-3 inches taller than me – imprecise because it’s so hard to get data from behind the Iron Curtain I guess.

So maybe it’s time for me to get in shape, huh?  I haven’t played hoops in ten years.  I’m about 30 pounds heavier than I’d like to be.  And my hoops shoes…  well, I bought those in 1996 and they still have tread on them.

So this weekend I began my training.  I found the biggest dude I know, my friend Mike, and we played a series of one on one games.  It was ugly – Mike’s taller than me, outweighs me by 60 pounds, and is all around just far more athletic.  I was lucky to steal one game from him, while he annihilated me three times.

Afterwards, we still had enough breath in us to role-play The Ugly Americans:

No Flopping

I thought this was kinda cool…

May 14, 2010

Check out what happens if you start to type “PowerPivot” into Google’s search box these days:

clip_image001

Very gratifying, for sure.  Thanks folks, this feels good.

(I won’t comment on Bing, which clearly favors the blogs of current MS employees – cough cough PowerPivotGeek cough cough)

What I’ve Been Doing, What I Will Talk About Next

Maurice Prather and I have been heads-down this week getting PivotStream’s server farm up to date with SharePoint and PowerPivot RTM.  Four servers working together as a unit.  It feels like a phenomenal amount of power and capability, to be honest.

Here are some things on the agenda for the next week or two, in terms of the blog:

  1. Follow-up on my usage of Notepad++ – thank you Colin, I will never go back
  2. An overview of our PowerPivot farm, tradeoffs we’ve made, considerations we covered, etc.
  3. Re-usage techniques for PowerPivot logic, and workarounds for PowerPivot’s lack of data-level security
  4. Our first online, interactive PowerPivot demos! – Yes, these are coming :)
  5. My upcoming Teched joint session with that villainous hijacker of search engines, the PowerPivotGeek himself, Dave Wickert.

There are a few other things as well that I am sure I have forgotten.  Can’t wait to share.  May have to tell the folks at PivotStream that I’m going dark for a day or two to just pump out blog content :)

Any preference on where I start in the list above?  Drop me a comment.  I’m happy to tailor the sequence.


Automating the “Site Visitors” XL Report

May 6, 2010

 
Don Corleone Evaluates PowerPivot 
“What’s in it for me?”

-Average Excel Pro

OK, the average Excel pro has not caught on yet.  Heck, most of them haven’t even heard of PowerPivot yet – it did just release this week after all.

Let’s say you don’t have much need for handling large data volumes, your organization isn’t large enough to worry about spreadsheet robustness/business intelligence/one version of the truth, and you haven’t yet seen what DAX can do for you in pivots.

Or maybe you get some of that, but don’t really understand this whole SharePoint thing.

If you are in either of those boats, here’s an example from my work this weekend.

My First Production, Scheduled SharePoint PowerPivot Report!

You know that chart of PowerPivot visitors by industry that I posted earlier this week?  For months now, I’ve been logging that stuff in an Excel workbook:

Site Visitors in Excel

Over time, it’s grown to 500+ rows.  No big deal. 

Well, OK, there are a few problems.

  1. Workbook editing on many machines.  I have a desktop, a netbook, and two laptops.  Guess which one always has the latest version of the workbook?  Yep, that’s right, whichever computer I am NOT using at the moment I want to edit the file.  So I get workbooks on every machine.  That leads to merge fun, duplicate records, and sometimes even lost records.  It also makes the process not a lot of fun, so I tend to neglect it at times.
  2. No time/date stamping.  I’m too lazy to enter a date every time, so the best approximation of “date first logged” is the sort order of the spreadsheet.  Not so precise, especially once I sort and accidentally save :)
  3. Sharing with others.  I like to keep my colleagues at PivotStream apprised of the nature of the visitors I get on my sites.  We’re always evaluating different technologies and I want to make sure they are constantly reminded of what a good decision it’s been to adopt PowerPivot :)   And, well, emailing workbooks around is also clunky.  (I won’t bore you with why it’s clunky, but ask me for details and I will oblige.)

Ideally, what I want is a central, unified copy of the list, that is also centrally editable, constantly timestamped, backed up, as easy to view as a web page…  and all with the editing ease of Excel.  No webforms, please.

Tall order.  But like Inigo Montoya, I may know something you don’t know.

SharePoint List With DataSheet View!

Have you ever seen a SharePoint list?  If you’ve visited the FAQ, you have, because, well, that’s just a vanilla SharePoint list.

Well, I transferred the list from Excel to a SharePoint list on our new PivotStream (internal) SharePoint 2010 farm.  With a few clicks in the web UI, I get the list looking quite spiffy:

Visitors List in SharePoint HTML View

SharePoint DataSheet View ButtonBut how did I get the data out of Excel and into the list?  That’s where DataSheet View comes in.  Here’s the SharePoint ribbon button you click to switch into DataSheet View, at right:

Once you click that button, you get an editing experience that is VERY similar to Excel or Access, as seen below:
  

DataSheet View For Bulk Edit

It’s no accident that it looks a lot like Excel or Access really, since this view was first built by a mixture of engineers from both of those teams about six years ago.  This is an ActiveX control that is installed with Office client, and it’s one of the hidden little gems of Office / SharePoint integration.

Anyway, using this view, I was able to do a bulk copy/paste of all 500+ rows from Excel, directly into the grid, and then they just streamed up to SharePoint in the background.  Took less than 30 seconds.

Now, when I want to add a new item, I come to this view, and start entering data as if I were in Excel.  I even get autocomplete :)   SharePoint adds a date/time stamp behind the scenes.

Now it Gets Fun:  Importing the List into PowerPivot

OK, great, now I have a place to edit my list that suits all of my requirements.  No more multi-machine, multi-workbook merge hassle.

But now that my list is stored on SharePoint, naturally, I want to show some charts in SharePoint!

Here’s another feature you may not have heard of:  every SharePoint 2010 list is now also a valid Data Feed source.  Why should you care?  Because PowerPivot LOVES Data Feeds.

Time to click another button in the SharePoint ribbon, “Export as Data Feed,” circled in green below:

Export as Data Feed Button

I get the usual series of helpful security dialogs.  Yay, I feel safe!  Doubly safe, since there are two dialogs, heh heh.

 Second Security Dlg First Security Dlg

Right now, PowerPivot is the only client app (at least on my computers) that is registered to handle data feeds, so after that, I get a dialog from the PowerPivot addin that asks me which workbook I’d like to add this data into, followed by the import wizard:

PowerPivot Data Feed Workbook Chooser Import into PowerPivot

I can also preview and filter the incoming data in the import wizard if I want :)   But I want all records, so I just click Finish, and the data gets imported into the desired XLSX, as data in the PowerPivot window:

SharePoint List Imported into PowerPivot

Building the chart at that point is of course pretty trivial.

Note also that I can refresh the source data by clicking the PowerPivot Refresh button at any time, since PowerPivot remembers where the data feed is located up on SharePoint.  So from time to time I can update my report to reflect the latest visitor statistics.

But why do this manually?  I’m too lazy for that.

Upload to SharePoint

Since our SharePoint 2010 farm is actually MOSS Enterprise with PowerPivot for SharePoint installed, I can upload my XLSX now and have it render on the server instead via Excel Services.

Here is the chart from the workbook, rendering as an embedded web part on our internal team SharePoint site’s home page:

PowerPivot Report Embedded as a SharePoint Web Part

As I said though, I don’t want that chart to be stale all the time.  I don’t want to have to upload a new file every time I update the data.

With PowerPivot, I can schedule the refresh.  Go to the report gallery view where the workbook was uploaded, and click the highlighted little calendar icon (circled in orange):

PowerPivot Gallery

Now I get this page that lets me schedule data refresh to happen automatically, like for instance, once a night:

PowerPivot Data Refresh Schedule Page

The first time you do this with a new workbook, I also recommend setting the “Also refresh as soon as possible” checkbox so you can verify that it worked:

PowerPivot One Time Data Refresh

Summary

So, there you have it.  My simple list didn’t have large data volumes, multiple tables, complex calc requirements, or any of those things we typically talk about when we discuss the strengths of PowerPivot.

But even so, the combination of PowerPivot and SharePoint turned out to be the best solution to my 100% real-life problem.  It even took me longer to write this blog post than to set it up :)


May 12th in NYC: “IT Can’t Hide from MS Excel”

May 5, 2010

Quick tip:  if you live in the New York City area, I highly recommend looking into this session on May 12th:

http://tdwichapters.org/Blogs/New-York/2010/04/May-12th-2010-IT-Can-Run-But-IT-Cant-Hide-from-MS-Excel.aspx

It features Bill Baker and Andrew Brust.  I worked with Bill Baker at Microsoft while he ran the Business Intelligence arm of SQL.  Very colorful guy with an illustrious history and a reputation for not pulling punches.

I’ve never met Andrew personally but he was one of the first people I “met” on Twitter, and I quickly came to respect his judgment and observations.  In fact he’s been on my blogroll (in the right sidebar) virtually since the day I launched this blog.

PowerPivot is not explicitly mentioned in the agenda, but I have word that it figures heavily in the discussion.  (I suspect that an explicit mention would make people fear a marketing talk, but neither of these guys is an MS employee so that wouldn’t be warranted really).


PowerPivot Interest By Industry

May 4, 2010

For the past few months, I’ve been tracking visitors to this site, as well as to the FAQ site, and periodically going thru and tagging them by industry.

I thought maybe everyone else would be interested, so here ya go, the top 15 industries:

PowerPivot Visitors by Industry Chart

Note #1:  this just reflects people I can recognize – for most visitors, I only see their ISP name.  And anyone who reads strictly over RSS or Email Subscription, I don’t see them either.  But it’s still a reasonable comparison of interest across industries.

Note #2:  this just reflects visitors from the U.S.  Given the amount of manual tagging work involved, I decided not to track the international traffic, which overall outweighs the U.S. by quite a bit.

Any surprises?  For me, here are a few:

  1. Healthcare – I had no idea that they were quite this embracing of cutting edge analysis/reporting technology
  2. Financial Services – yes, this makes total sense.  But given my perception that most visitors so far have been from the Business Intelligence crowd and not the Excel pro crowd, well, this makes me second guess that assumption.
  3. Government – wow, you wouldn’t believe the names I see.  From City/County level all the way up thru Federal.

PowerPivot v1 Released to MSDN!

May 3, 2010

It’s been a long time coming, but worth the wait.

No more beta, folks, the final v1 bits are now released on MSDN:

https://msdn.microsoft.com/en-us/subscriptions/securedownloads/default.aspx

The client addon is listed under “PowerPivot” and the server is under “SQL Server 2008 R2”

Remember that you will also need the final released version of Office 2010 to install the addon, and the final version of SharePoint 2010 to install the server.  (I suspect the RC version may suffice for either of those as well, but definitely NOT the Beta 2 releases from the Fall).

For non-MSDN subscribers, I also suspect that powerpivot.com will be updated shortly with public links to the client addon.