For some time, I have been looking around for a fairly complete date table in Excel for use with PowerPivot. If you are working with data derived from a data warehouse, a date table is perhaps the most common dimension table that exists in the warehouse. However, not every scenario involves working with a data warehouse directly, and I simply wanted a “portable” date table. I found very little online, the best perhaps being this Excel table offered by the Kimball group (the table has been expanded since I originally downloaded it). I could have modified the Kimball table for my particular needs, but I decided to create one from scratch.  Late last year, Rob posted an article titled the Ultimate Date Table, which is available from the Azure Marketplace. I considered using this table instead of the one I was building in Excel, but the “Ultimate Table” lacks fiscal periods. Much of the analysis work I do includes fiscal periods.

For the Excel table that I was building, I checked the validity of some of the date period formulas I created by using date tables generated from the Date Dimension Wizard in Analysis Services. In the process of generating these tables,  a thought struck me – why not duplicate the functionality of the Date Dimension Wizard using nothing more than Excel formulas? Hence, my  goal changed from from creating a Regular Date/Fiscal Date table to one that included the additional calendars used in businesses. If you are an independent consultant with a varied clientele , you cannot anticipate what calendars a client will need, so I decided to cover all bases. For those of you that are unfamiliar with the Date Dimension Wizard in Analysis Services (only available in Multi-Dimensional mode), Figures 1a & 1b show the first two steps of the wizard.

Fig 1a – Date Dimension Wizard Step 1

Figure 1b – Date Dimension Wizard Step 2

As you can see from the above figures, you can generate up to five different calendars. Adding periods for the ISO calendar in the Excel date table was straightforward, but I came to a grinding halt when I turned my attention to the reporting calendar. The problem with the reporting calendar was that I couldn’t find any consistent definition of this calendar. I checked for information on this calendar online, and I contacted some folks familiar with the 4-4-5 pattern of the calendar. However, there was no consistency in the information I gathered. At this point, I put the reporting and manufacturing calendars on hold, and used my new table with the other three calendars. Every so often, I returned to tackle the reporting calendar, but to no avail. Eventually, I realized that I was using the wrong approach for creating this calendar. Since the Dimension Wizard obviously generated reporting periods using well defined criteria, all I had to do was “reverse engineer” the generated results. Using this approach, I was able to complete the reporting and manufacturing calendars without further issues. Figure 2 shows the configuration options in the final version of the Excel date table. This entry form was created on a worksheet separate from the date table.

Figure 2 – Excel Date Table Configuration Options

In addition to the options available in the Dimension Wizard, I have included the ability to define weekend days, and one or more holiday tables. Another option (not shown) is the ability to create simple “selling season” periods. The actual date table is totally configurable – change the column titles and text that appears in the columns to suit your needs (including using a different language). Add or remove date period columns, change the year window of the calendar (it’s 21 years by default), and so on.

Anyone interested in using the date table can download it here. The workbook includes detailed information and instructions regarding the use of the table.

You may recall in my last post, COMMISSION CALCULATIONS IN POWERPIVOT, we got to the point where we could dynamically calculate the sales value and attributable commission rate that should be applied based on time, value and team parameters, reading from a Rates table.

In this post, we’ll complete the commission calculation, providing a different value for individuals and the team manager.

Where are we?

Just to recap, we got to the point in my last post where we had calculated [Sales_Value] and [Comm_Rate] as below:

It seemed logical that the concluding element was to simply multiply one value by the other.  To a degree that is correct, but unfortunately, it’s not quite so straightforward.  Let’s take a look at what that would do with a simple measure that I’ll call [Comm_V1]

Comm_V1 = [Sales_Value]*[Comm_Rate]

You’ll notice that the measure provides the right calculations, but only at the level of the report where it finds a single People[Name] and Dates[MonthEndDate].  There’s no aggregation above that value but why?

I’ll try to explain.  Our [Comm_Rate] measure only calculates when the following condition holds true

IF(COUNTROWS(VALUES(Dates[MonthEndDate]))=1 

    &&COUNTROWS(VALUES(People[Name]))=1

    &&COUNTROWS(VALUES(Types[Type_Code]))=1 ……

This means that we will see BLANK() returned as a [Comm_Rate] when we have more than one Dates[MonthEndDate] or more than one People[Name] or more than one Types[Type_Code].  Our [Comm_V1] measure is therefore trying to multiply the subtotal of [Sales_Value] by BLANK() which will obviously return a blank.

However, check out what happens when I only have one Dates[MonthEndDate]:

As you might expect, we get nothing back.  But look what happens if I open up one of the People[Name] fields:

Bizarrely, I don’t just have values for the name that I’ve opened up, but I also have results for everyone else.  And, the results are correct.  However, it’s useless to us as it doesn’t work with more than one date and I’m pretty sure that this other phenomenon is just a bug.

The Real Solution – Bring on SUMX

There’s been a number of brilliant posts on SUMX such as THIS ONE.  In short, SUMX is an iterative function. The syntax for SUMX is SUMX(table, expression).  What this means is that SUMX will process the “expression” for each item in the “table”.  However, it then goes one step further.  SUMX will then SUM the results at each aggregation level in the report.

We know that our “expression” has to be [Sales_Value] * [Comm_Rate] and we want to complete this equation for each Dates[MonthEndDate] because, as mentioned earlier, this is the only level in our report where we can calculate a [Comm_Rate].  We want the outcome of those equations to be added together through the aggregation levels in our report such as People[Name] and People[Manager].

Let’s try that with a measure called [Comm_V2] (I apologise for not being very inventive with these names – you’ve probably guessed from the temporary nature of these names that I’m ultimately going to ditch them )

Comm_V2

=SUMX(VALUES(Dates[MonthEndDate]),[Sales_Value]*[Comm_Rate])

Before I proceed with showing the results of this measure, let me explain why I’m using VALUES(Dates[MonthEndDate]).  There’s actually two reasons:

1. MonthEndDate is a field on my Dates table.  I could use Dates as my table, but I would then be evaluating my expression for every single date in that table.  Not only is that inefficient, but it will inevitably give me the wrong answer.
2. In order to make SUMX as efficient as possible, you need to present it with the simplest and smallest number of distinct elements.  By using VALUES(Dates[MonthEndDate]) I’m creating an “in memory” list of distinct MonthEndDate values.  Our expression will be evaluated once for each of these distinct values and we’re not holding a load of excess column baggage in memory.

This looks pretty reasonable.  The individual monthly commission values are adding up to a correct total for each person.  However, is Dalglish’s value correct?  I thought this would be the sum of everyone’s commission within the team.  If the hidden people are shown and we add up the column, the value for our total of Audio_Visual should be 98,989.18.

So what is this “Manager’s Total” doing?

It would be reasonable to believe that the underlying elements for Dalglish that should be aggregated are the team members (People[Name]) and the Dates[MonthEndDate].  However, we haven’t told PowerPivot to do that.  The report is laid out in that manner, but all we’ve told PowerPivot to do is create an aggregation based on Dates[MonthEndDate].  Therefore, PowerPivot has added together the monthly totals for Dalglish as an individual.  A breakdown of the equation is as below:

So, it’s actually calculating the Manager’s commission.  What if I want it to total the sum of commissions paid to the whole team?

Nested SUMX

The simple answer to this issue is that we have to also tell our measure to aggregate for People[Name].  SUMX only provides for one iteration level.  I don’t know the technical reasons as to why that is, but I imagine that providing multiple iterations could present problems with solve orders amongst other things.  However, we can nest a SUMX expression inside another SUMX expression.  The solution is as follows in a measure that doesn’t have a temporary name (which means it’s right – we hope) – Team_Commission

Team_Commission

=SUMX(VALUES(People[Name]),

SUMX(VALUES(Dates[MonthEndDate]),[Sales_Value]*[Comm_Rate])

)

I want to evaluate our previous SUMX expression for each distinct People[Name] value.  Therefore, I use VALUES() to get a distinct list of items in that field and the expression is executed for each member and then aggregated.

Bingo – we have the right values.

Separate Manager’s Commission

Although our Team_Commission measure is accurate and we don’t have any spurious totals that could serve to confuse, the previous value that we had for Dalglish did make sense in the right context.  That was the value that he could expect for commission.  It therefore makes sense to compile a further measure that is computed just at the Manager level to show that commission value.  It’s a variation on a theme so I’ll jump straight into the DAX

Manager_Commission

=IF(COUNTROWS(VALUES(People[Name]))>1,

SUMX(VALUES(People[Manager]),

SUMX(VALUES(Dates[MonthEndDate]),[Sales_Value]*[Comm_Rate])

),

    BLANK()

    )

In this measure, we’re iterating over a distinct list of People[Manager].  You’ll notice that I only want this to happen where COUNTROWS(VALUES(People[Name]))>1.  I’m using that to determine the “Manager Level” in our report.  An individual salesperson would evaluate to 1 in this measure whereas a manager will evaluate to however many individuals in their team which we’re assuming is greater than 1.

Hang on – isn’t that a bit rash!  Can a manager have only one report?  In theory, I guess that’s true.  This is my biggest issue with the way that PowerPivot works with report totals.  I’ve only got COUNTROWS in my armoury to deal with this.

Fortunately, there’s a way around it!   The answer is to simply add the Manager into the People table and have them report to themselves.  I’ve adjusted the table to show Robinson reporting to “Someone Else”.  I’ve then created records for “Someone Else” and “Dalglish” reporting to themselves.

Now my report can handle the fact that a team may only have one member, but it stills manages to understand that my measure has to execute.

And Finally…

I said to Erik in the comments section of the first post in this series that I would show how to calculate commission where the increase in percentage only applies to the element over a certain value.  Unfortunately, there’s not time to do this here, but I will come back to it.  Sorry that I couldn’t get that in this post Erik.

Why yes.  Yes we are.  More effectively every day in fact.

Firstly, I have to be clear that I’m not presenting a “one-size-fits-all” approach to sales commission calculations here.  That wouldn’t be possible because commission schemes vary extensively and, in my experience, some schemes aren’t even based on a logic that can derive a mathematical answer!  However, I’m presenting an approach here that will hopefully provide a template approach that can be modified to a number of different scenarios.

The Target Outcome

The scheme that I’m using in this example operates as follows:

1. Commission is paid monthly based on the achievement in that month
2. As a salesperson sells more, then accelerators trigger.  That is to say, for example, a salesperson may receive 1% of sales up to £10,000 and 2% of sales between £10,000 and £20,000.  The 2% is payable on the whole value.  Therefore, if a salesperson sells £11,000 then they would receive 2% of the full £11,000
3. Percentages and bands can change monthly at management discretion
4. This is a monthly threshold so each salesperson is reset to zero at the start of each month.
5. Each product group carries a separate set of bands and rates
6. Managers receive commission at a different rate to the sales team based on the total sales for their team.

Naturally, this is a hypothetical dataset and commission scheme but it is closely based on a real life example that I implemented recently.  I’ve hacked together a dataset as I couldn’t release my real life example for obvious reasons.

The Dataset

Before I go into the schema, it’s worthwhile noting once again that I have hacked this dataset together.  Because I lack the imagination to dream up names for members of the sales team and also because I was just reading a write up on Liverpool’s drubbing of Chelsea yesterday, I’ve used the Liverpool squad reporting into “King Kenny”.  Obviously, I’m not implying for one minute that my beloved Liverpool team should turn their hand to selling AV and Computer electricals!

Note – I’ve compiled this workbook in PowerPivot V1 but quickly whipped it through V2 to get the nice table view above!

I have four related tables

1. Orders– this is a list of orders detailing the usual candidates for a dataset like this – Order (being the order number), Date, Salesman, Qty, unit_Cost, Value and Type.
2. People– this is a list of the unique salesperson names and the manager to whom they report.
3. Types– this is a product grouping.  As my commission scheme carries different parameters and rates dependent on the product group, I have to be able to analyse my dataset at this level.  In this case, I have two product groups being AV (Audio Visual) and COMP (Computers)
4. Dates– this is a fairly standard dates table.  I have a sequential list of dates with a number of attributes.  In truth, most of these attributes are redundant for this analysis.  The main attribute is the MonthEndDate which I will use in the DAX measure that we create.
5. Rates – this is NOT RELATED to any other tables.  The reason for this is because I want to call in information in different sections of this table that, in turn, is dependent on the outcome of a dynamic measure.

The Dax

A Simple “Starter for 10”

Obviously, we need to calculate how much has been sold by an individual salesperson in a given month in order to then calculate how much commission accrues as a result.  My final analysis is going to be carrying People[Name], People[Manager] and Dates[MonthEndDate] on rows in my report.  As a result, I don’t have to worry about applying any cunning filters.  The upshot is that we can resort to a simple SUM() measure.

Sales_Value

=SUM(Orders[Value])

Rob – is there a level 0 on the spicy scale?

And then it gets a “bit” harder”!

Whilst Sales_Value may not be the most intricate DAX expression you’ve ever seen, things get a bit more tasty when we try to determine which percentage rate to apply in our calculation.  I’ll dive straight into the DAX.

Comm_Rate

=IF(COUNTROWS(VALUES(Dates[MonthEndDate]))=1

&&COUNTROWS(VALUES(People[Name]))=1

&&COUNTROWS(VALUES(Types[Type_Code]))=1,

     IF([Sales_Value]=BLANK(),BLANK(),

       CALCULATE(MAX(Rates[Rate]),

          FILTER(Rates,

             Rates[From]<=[Sales_Value]

             &&Rates[To]>=[Sales_Value]

             &&Rates[From_Date]<=MAX(Dates[MonthEndDate])

             &&Rates[To_Date]>=MAX(Dates[MonthEndDate])

             &&Rates[Type]=VALUES(Types[Type_Code])

             &&Rates[Rate_Group]=”Salesperson”

                    )

                    )

           ),

    IF(COUNTROWS(VALUES(Dates[MonthEndDate]))=1

   &&COUNTROWS(VALUES(People[Name]))>1

   &&COUNTROWS(VALUES(Types[Type_Code]))=1,

          IF([Sales_Value]=BLANK(),BLANK(),

              CALCULATE(MAX(Rates[Rate]),

          FILTER(Rates,

             Rates[From]<=[Sales_Value]

             &&Rates[To]>=[Sales_Value]

             &&Rates[From_Date]<=MAX(Dates[MonthEndDate])

             &&Rates[To_Date]>=MAX(Dates[MonthEndDate])

             &&Rates[Type]=VALUES(Types[Type_Code])

             &&Rates[Rate_Group]=”Manager”

                         )

                        )

             )

       )

)

Let’s break that down.

We only want to evaluate a rate when:

1. We have one MonthEndDate by using IF(COUNTROWS(VALUES(Dates[MonthEndDate]))=1. It doesn’t make sense to calculate at any other point through the aggregation.  If we were to do so, the fact that we’re using MAX(Rates[Rate]) would mean that the highest value over a combination of months would be returned at an aggregation level where multiple months were included.
2. We want to calculate a rate for each salesperson independently.  To do this, we add a further condition &&COUNTROWS(VALUES(People[Name]))=1.
3. We have different rates for each Type (where Type is the product group – in this case Audio Visual and Computers)
4. We only want to return a value when we have a Sales_Value.  This is more to do with aesthetics than anything else.  If you have a Salesperson who doesn’t sell anything or wasn’t perhaps employed at a point in time of the analysis, you’ll get a series of unnecessary additional lines.  We therefore exclude those situations using IF([Sales_Value]=BLANK(),BLANK().

Where these conditions hold true, we want to pick the appropriate rate.

The Rates Table

Before we get onto the FILTER() expression, it probably makes sense to take a good look at the Rates table to be clear on what we’re looking at:

Remember that this table ISN’T related to any of our other tables.  As a result, we have to provide all of the conditions that will filter this list down to one result and we can then return the Rates[Rate] value.  As a result, we need to plot the conditions below within a FILTER() condition:

1. Our [Sales_Value] must fit between the Rates[From] and Rates[To] values.  You’ll notice that the values in this rates table follow on from one another so we should be able to guarantee a result.  This is done using Rates[From]<=[Sales_Value]&&Rates[To]>=[Sales_Value]
2. We need to provide a value for Rates[Type] as we could conceivably return multiple results for multiple products and our measure will therefore have to evaluate a MAX() result.  This is done using &&Rates[Type]=VALUES(Types[Type_Code])
3. We can return different rates for different periods in time.  The Order can be pigeon holed between Rates[From_Date] and Rates[To_Date] using &&Rates[From_Date]<=MAX(Dates[MonthEndDate])&&Rates[To_Date]>=MAX(Dates[MonthEndDate])
4. Finally, because we can get a different rate for a Salesperson from a Manager, we need to explicitly state that &&Rates[Rate_Group]=”Salesperson”.  You’ll recall that our upfront IF() statement has already determined that we’re evaluating at the Salesperson level using &&COUNTROWS(VALUES(People[Name]))=1

Having done all of that, we then do the same again but for Managers.  The only difference is that we evaluate when &&COUNTROWS(VALUES(People[Name]))>1  to determine that we’re at the manager level (based on a Manager having more than one Salesperson reporting to them) and &&Rates[Rate_Group]=”Manager” for obvious reasons.

The outcome with these measures in place shows the following:

So what next?

Surely, we can now calculate commission using [Sales_Value]*[Comm_Rate] right?  If it were that easy, I wouldn’t be lining up a parting comment to say that I’ll be back with a follow up to show how it’s done.  To whet appetites, think nested SUMX!

Thanks for All the Emails!

I really appreciate everyone who took the time to send in their thoughts on the first iteration of the ROI Quadrant.  Your input has encouraged me to keep refining and updating the chart. 

Based on your feedback, there are now a few additions in the version above:

1. Desktop-Only PowerPivot
2. QlikView
3. Birst
4. PivotLink

I’ll briefly explain the rationale for each.

Desktop PowerPivot vs. “Full System” PowerPivot

This is a distinction I often fail to make and you guys rightly called me on it.

Let me be clear:  PowerPivot on your desktop is an amazing tool, but it does not offer the ongoing robustness benefits that you get when you deploy it as part of a full system.

What does the full system entail?  Some of it isn’t software or hardware, but a social system.  People in different roles cooperating.  How your toolset blends in with your people and your organization is important – another theme I keep emphasizing.

1. PowerPivot for SharePoint.  Without this, you don’t get security, auto-refresh, one version of the truth, cross-platform reach, nor the simpler usability of the web interface.  This is a big reason why people come to us for Cloud PowerPivot – to gain these benefits without large IT investments.
2. Database Support
1. I’ve spoken at length about how much better PowerPivot is when paired with a database – here and here for starters.
2. ETL technically means “extract, transform, and load” but in general I just use this to refer to data shaping – something that is difficult (often impossible) to perform in PowerPivot.
3. Even more clearly though, PowerPivot authors can deliver much better results when they have the cooperation of a database pro.  A two-person team of PowerPivot “Excel Pro” plus a “Database Pro” can accomplish things that much larger teams will actually never accomplish using other tools.  Whenever I conduct one of my two-day training sessions on PowerPivot, this cooperation is a theme that I stress repeatedly.

If you’re just running PowerPivot on the desktop, you still get dramatically increased robustness and agility over regular Excel, thanks to portable formulas and multi-table mashups.  But the lack of DBA support reduces your agility quite a bit compared to adopting a complete system, and the lack of SharePoint impacts both your robustness and agility.

The difference between having a full system versus just the desktop is night and day.  If all you have right now is the desktop version, don’t sweat it – you are learning all the right things and gaining tremendous benefit already.  Look at this as a positive – there are still further amazing capabilities to leverage.

QlikView

I’ve heard a lot of good things about QlikView over the years and that continued in the emails I received.

Basically the theme I saw was in line with my long-simmering theory:  if you are in the market for a BI tool, but want something a bit more approachable than traditional BI platforms, QlikView is a very appealing choice.  The feature set is very rich, quite competitive with “big box” BI vendors, while presenting to its users a level of complexity that resembles that of Access development rather than heavyweight BI.

Still, I’ve met a number of people who who’ve been using or reselling QlikView and decided that it wasn’t broadly-useable enough.  The scripting language can put people off, and there’s a rich community of QlikView consultants, which is another sign of complexity.

Again, that theme that it’s like Access development in terms of complexity and less like Excel.

So QlikView gets a larger dot than traditional BI, similar robustness level (greater than PowerPivot), but a smaller dot than PowerPivot and lower agility score.  The Excel Army has soldiers everywhere, ready to engage.  To adopt QlikView you need to teach people a brand new language.

Note that if I did start to include feature depth using color intensity, QlikView would do very well on that scale.  But features don’t drive ROI nearly as much as the way it fits the people in your organization, so PowerPivot’s “ROI Rectangle” would still be quite a bit larger:

Comparing the “ROI Rectangles” of Traditional BI, QlikView, and PowerPivot

Birst

I know very little about Birst personally but I received a few suggestions in terms of where to place it, and those suggestions were consistent, so here it is.  Let me know if you disagree.

PivotLink

At Pivotstream we have a great deal of experience with PivotLink.  Or at least, my colleagues do.  My acknowledged first mission when I joined the company was to replace PivotLink with PowerPivot.  We did that quickly and we never looked back.

I realize that my placement of their dot is inconsistent with their marketing, as they were one of the early “self-service SaaS BI” firms.  Good ideas and courage are excellent ingredients, and I salute them for that, but we have firsthand knowledge that the execution is lacking.

I don’t have a problem being particularly harsh here because the notion of friendly BI tools is very much a reality – I live it every day.  So anyone that purports to be that but then under-delivers on usability AND robustness could unfairly give the entire space a bad name.

Tableau?

Tableau is a very popular tool in the data visualization crowd.  Those of you who have used it as well as PowerPivot, please drop me a note with your thoughts.  Your anonymity will be protected.  I’m rob.  At a place called Pivotstream.  Dot com.

Two mini-posts today:  this one and then an updated ROI Quadrant.

This one falls under the “quick tip” category (and the “MacGyver” category were I to have one).

How “Up to Date” Is Your Data Source?

Sometimes you’ll find yourself wanting to know the absolute latest date in your data, so that you can calculate things like “how much of the current month has already elapsed” for instance.

And it’s tempting to use the TODAY() function to do that, but oftentimes your data “lags” behind today – maybe by a day, maybe more.  So TODAY() becomes pretty worthless for that.

So, what do you do?  I’ve recently started using a simple trick that I like better than my past approaches, so I thought I’d share it.

First, Decide on Calendar vs. Sales (or Other “Fact”) Table

If you’re lucky, your Calendar table is only as “up to date” as the rest of your data.  So if you only have sales transaction data through May 1st, May 1st is also the most recent date in your Calendar table.  If that’s the case, I recommend using the Calendar table as the basis of your solution.

If your Calendar table extends beyond the most current date, then you have to use one of your data tables like Sales as the basis.

Then there are a few different approaches you can take.

Worst Technique:  An ALL() Measure

I call this the “worst” but I admit that I’ve done it a lot.  You write a measure:

[CurrentAsOfDate]
=CALCULATE(LASTDATE(Sales[OrderDate]), ALL(Sales))

And then you use that measure in your other measures, to calculate things like [% Month Remaining].

The trouble with this is that it has to recalc all the time, every time you manipulate a pivot, which is wasteful since it is always going to return the same value.  It has to strip filters from the Sales table, and then scan the OrderDate column (which could be millions of rows) for the max date.  It might not be a lot of work for the PowerPivot engine, but why force it to do anything extra?

Clunky Technique:  Calc Column in Sales, Then a Measure

First You Add a LASTDATE(ALL(Sales[OrderDate])) Calc Column to Sales Table

OK, then you add a measure much like before:

[CurrentAsOfDate]
=LASTDATE(Sales[MostRecentSale))

That’s better – this measure still has to be evaluated every time you update the pivot, but there is no ALL() that needs to be applied, so that’s less work.

But still, we can do better.

MacGyver Says:  One-Row Table!

Even better.  Just go into Excel and create a one-row “table” like this:

Just Type This in,
Then Copy to Clipboard

Copy that to the clipboard, then switch to PowerPivot and Paste:

Paste as New Table

Then you add a calc column in THAT table, that takes the LASTDATE() of the Sales table:

Put Your LASTDATE() Calc Column in Your “MacGyver” Table

And THEN you write your measure:

[CurrentAsOfDate]
=LASTDATE(MacGyver[MostRecentSaleDates))

That is about as efficient as it gets.  No ALL(), no rows to scan, no “pollution” of your large Sales table, less impact on file size.

It Updates Every Time You Refresh Your Sales Table…

…which is exactly what you need – no more or less frequently.

And hey, the economy is numbers-related so it’s not entirely off topic for this blog right?

Nobel Economist Paul Krugman Wants More “Stimulus Spending”

Stimulus spending is generally Mr. Krugman’s preferred fix for the US (and world) economy.  Lately, he hasn’t been getting enough of it so he is on tour these days demanding more.

What exactly IS stimulus spending?  Fundamentally it means “the government spends or gives away money.”  The idea is that by injecting new money into the economy, it can kickstart a benevolent cycle that sustains itself:

1. The people or businesses who receive the stimulus cash now have more money to spend on other things.
2. As THEY spend their newfound money, other businesses have to hire more people to fill the demand.
3. Those newly-employed workers now have more money to spend, and may even end up buying from the people who received the stimulus in step one.

Where Does the Government Get the Stimulus Cash?

It’s hard to argue with a fresh injection of money into the economy, and I highly recommend that the government should let ME play a big role in step #1 above.  I am a willing conduit for your good works Paul, and honestly, the bigger the check the better.  I can do a lot more good for the economy with, say, $100k than I can with $1k.  I promise I will spend every penny in short order, no hoarding. 

But as a society we really do need to ask where the money comes from in order to feel good about it.  By my accounting, it can only come from a handful of places:

1. Money that the government has saved
2. Money that the government refunds directly to businesses and taxpayers via tax credits, rebates, or cuts
3. Money raised by new taxes
4. Money that the government just flat-out prints
5. Money that is borrowed:
1. From other governments
2. From private corporations
3. From banks, mutual funds, etc.
4. From private investors
5. From our own Federal Reserve

#1 kinda makes you laugh doesn’t it?  Governments with saved money?  Har har.  Certainly not the US government.  Does ANY government have money saved up?  Assets exceeding liabilities?  I need an accountant!  Where’s David Churchward?

Anyway let’s take a look at the other sources.

#2 – Tax Refunds/Credits/Cuts

Again, allow me to be first to volunteer to have some of my tax dollars returned to me, and the more the better.

Jokes aside, this isn’t such a bad idea in general, but it has consequences.  By taking in less money, the government now has less to spend on other things.  And reduced spending on other things is really just REDUCING stimulus in other places – if they lay off federal workers for instance, or buy less Predator drones, the federal workers or the defense firms will have less money to spend into the economy, and we’re back where we started.

Tax refunds really just transfer “stimulus” from one place to another.  So by itself, this one is no help.

#3 – New Taxes

Take more money out of the economy by raising taxes and then distribute it as stimulus = just another way of moving money around.  Anti-stimulus one place, stimulus in another.  Yields no net gain in stimulus.

#4 – Newly Printed Money

Now we’re getting somewhere.  If we’re going to have some real economic kickstart without just moving money from one place to another, conjuring new money out of thin air is a pretty good way to source it.  And the government, with the cooperation of the Federal Reserve, can do just that.

They’ve done a sizable amount of it lately, too.  They don’t call it money printing though, they call it “Quantitative Easing.”  (There’s a really funny video here too).

There are definitely real “ripple effects” from money printing that are not so good:

1. If they increase the money supply by 10%, that reduces the purchasing power of the dollar by 10%, because supply and demand applies to dollars too, and prices of goods rise by 10% on average in response to more dollars in existence
2. In other words, your dollars in the bank are now worth 10% less
3. Similarly, the wages you receive from your job (or Social Security) are worth 10% less
4. Even your investments (stocks, mutual funds, etc.) are worth less, unless they too appreciate by 10% as the money supply expands (which, actually, is reasonably likely)
5. This is called inflation

So money printing is really just another form of taxation.  It’s just a lot harder to connect the dots though, it’s a lot stealthier.  Gasoline prices are higher?  Blame the oil companies.  Blame speculators.  Just don’t blame the government’s stimulus packages ok?

Side Note:  “Reverse Robin Hood”

Krugman’s book and blog are named “Conscience of a Liberal.”  Some of his Nobel-winning work focused on rebuilding Haiti.  He talks a lot about egalitarian redistribution of wealth to the average citizen.  I don’t doubt that he’s sincerely trying to help people.

(And I, by the way, have no political affiliation – I don’t have a team that I root for, I actually think both of our teams here in the US are doing an equally awful job.)

So I wonder how he “squares” his egalitarian goals with these two facts:

1) Wealthier citizens have a much easier time shielding themselves from inflation

More sophisticated investors always do a better job outpacing inflation than the average citizen.  So the average citizen bears a disproportionate share of the “costs” of money printing.  The rich get richer, while both the poor AND the middle class get poorer.

2) Newly-printed money tends to enter the economy through the financial system

It’s a long story, but most newly-printed money goes to Wall Street before it goes anywhere else.

So again, money printing benefits the richest segment of society disproportionately – sure, their dollars are worth 10% less just like us, but their well-placed investments mitigate that, AND they receive the new money first.  So their net wealth, in dollars, increases by more than 10%, and they actually come out ahead.

The rest of us pay the bill.  This is so plain to me after a few years of reading that I wonder how it escapes him.  Or maybe I am missing something crucial.

#5 – Borrowed Money

Here’s another way to “create” new money without just moving it around.  Borrow it from someone else.

But actually that doesn’t create money either.  It’s another transfer.  In order to lend money to the government, I have to have excess money lying around.  I COULD use that money to just buy goods and services, and that would have the stimulating effect that Krugman desires.

But instead, I loan it to the government (you hear about this all the time – it’s called “buying Treasuries” or “buying T-bills”) so THEY can take care of the stimulus distribution.  It’s just another transfer within the system, no net gain in stimulus.

But there’s a catch:  it’s a loan, with interest, and the government then owes me back more money than I gave them.  Eek.

Even better:  the people lending the most money to the government again tend to be the wealthiest individuals and institutions.  So not only is borrowed money a “net zero” transfer within the system, it again disproportionately enriches the already rich.

Borrowing from other countries is a good trick in the sense that it is NOT a transfer within the system – money IS coming in from outside the system if it’s coming from China, as long as you view the US as a relatively closed system (which is at least partly true).

But since we have to pay it all back, with interest, we’re really borrowing from our own future.

Of course, there’s always the chance that we will NOT pay it back, as driven home by yet another hilarious video.  Then it truly is a Good Deal… for us.

“Hey Rob, you got a problem with the rich?”

Hell no.  I’ve been working harder these past couple of years than ever before in my career, and it’s not just because I enjoy it.  There are incentives for building a business from scratch like we are doing, if we are successful.

I actually think the whole “99% versus 1%” thing is quite misplaced.  We should all be striving to BE the 1% – I’m not there yet, and maybe never will be, but I sure won’t feel guilty for trying.  The things we do at Pivotstream are quite productive and helpful to our customers.  It’s easy for me to trace our benefit to those around us, and if we stopped benefiting them, that would be a bad sign for our business, as it should be.

I think a better framing of the battle lines would be the “99.99% versus the 0.01%,” or actually more accurately, “earned income versus unearned income.”

Look, Bill Gates and Steve Jobs changed the world emphatically.  They truly earned their billions.

Similarly, I think “real” investors like venture capitalists also earn their money, as covered in my Thanksgiving post from two years back.

But money that is made purely in financial markets isn’t earned in the same way as money earned via those other methods.  Financial market activity is a necessary lubricant, but then again so is the legal profession.  If we woke up one day and found that the legal profession was nearly 15% of our economy, as is the case with Wall Street, we’d immediately understand that something was wrong.

And even better:  unearned income is taxed at a much lower rate than earned income.  Sometimes it’s not taxed at all.  Warren Buffet paying less taxes than his secretary is a sign of a diseased system, one that incents the wrong things (creation and defense of wealth without proportionate benefit to society).

What’s wrong with a loan, really?

Businesses take loans all the time to get off the ground, or to expand operations in some significant manner.  And society DOES benefit from that real growth.

So is it the same thing for the government to do that?  Yes and no, is my answer.  No because governments don’t typically allocate investment as effectively as the private sector, but yes because if they DO invest properly, sometimes the benefits can even be GREATER than if the private sector did it.

Common infrastructure like the interstate system for instance is not something the private sector ever would (or could) have built.  Investments like that enable the private sector to then do things that never would have been possible before.

There certainly is a need for renewed investment in our infrastructure in the US (alternative energy and transportation come to mind, as well as improvements in education and job training).  Have you noticed any big projects on the scale of mid-twentieth century investments like the interstate system or the space program?  Yeah me either.

The other problem with loans:  exponentially-growing debt

If you borrow from time to time in order to expand your company, and then the new productive capacity of your company lets you pay off that debt, great.

But if you start to use debt in an endless cycle, and you become dependent on it all the time, it will crush you.

Check out our US government debt:

It took over 200 years to amass our first $6T in debt, then about 8 years to amass the second $6T, and we’re on pace to continue.

My question to Krugman on debt, then, is “if this isn’t enough debt, how much IS?  Would another $1T in loans and stimulus save us where the last six to ten TRILLION has not?”

“Follow the goods and services, not the money”

I think this is the most useful thing I have ever heard from an economist, but it wasn’t Krugman (I forget who, it was years ago).  We get too distracted by money, which is really just a medium for exchange, a lubricant, for REAL economic activity.  Real activity is all that truly matters, because money is worthless paper if you don’t exchange it for something real.

Take the example above:  stimulus spent on infrastructure can enable further productive activity that was impossible before.  Productive investments like that increase the size of the overall pie.

By contrast, stimulus money that ultimately gets spent on new clothes, or a hardwood floor, or sports tickets, is just re-allocation of “wealth” within a pie of a fixed size.

It’s the same with the earned/unearned income thing.  Earned income is productive, unearned income is largely the art of siphoning wealth from the productive.

“But Krugman Has a Nobel in Economics, Rob Has a Tech Blog”

Fair point.  I feel a bit weird about saying that this dude doesn’t know what he’s doing.  But I really think he doesn’t.  So why do I think someone that smart and that decorated can be so wrong?

Too Much Greek in the Math, Too Much Ivory in the Tower

If you crack open one of Krugman’s papers you are bound to see stuff like this:

Hey that’s fancy.  This is a big part of the world he lives in – that paper is half equations.  Go check his wikipedia entry – the man has never left academia.  On matters of the economy, why we would ever listen to someone who has never “tasted” anything to do with real business, but instead wraps himself in equations all day long, is beyond me.

It’s ironic for me, former Calculus Team captain in high school and proud owner of a number-crunching career, to make fun of someone for loving them some equations.  I debated grad school for awhile and almost went – if so, today I might be a lot like Krugman (only less decorated than he, and many times more bitter toward him).

But in my real-world experience with any form of complex system, the more “advanced” your equations become, and the less tempered they are by tangible experience, the more they diverge from reality.  If we were interviewing him for the job of economic adviser, his ability to wow us with equations wouldn’t be enough.  We’d want real answers to the sorts of questions above.  And given that his Nobel was awarded by a committee who similarly has lived in “isolated equation land” their entire lives, you start to realize that this could very well be a subculture that is completely alienated from anything real, and simply rewarding one of their own.

Too Little Friction in the Physics

You see the part in the image above where he says “this can’t happen, so..”

The paper is littered with stuff like that too.  That, and “taking as a given,” and “obviously,” and other ways to describe critical assumptions.  Papers like these are built on pyramids of those, and if any one of those assumptions proves to be untrue, even for a very brief period of time, it can all crumble down quickly.

It reminds me of high school physics, where half of the problems we had to solve started with “assume friction is zero” or “assume air resistance is negligible” or my personal favorite, “an object is moving through a vacuum…’

Great way to learn the theory, but you eventually have to deal with the dirty, noisy, friction-equipped real world.

(Size of circle = Usability and “Reach” of the Toolset)

Time to Evolve a Longstanding Meme

If you’ve been around Business Intelligence for the past ten years, you’ve seen a LOT of Gartner’s Magic Quadrant.  You can see the 2012 version here on Microstrategy’s web site.  (Microstrategy did very well on this year’s edition, so it makes sense that they display it).

The Magic Quadrant isn’t a bad tool for evaluating BI Vendors.  I like anything that breaks us out of 1-dimensional thought (I’d love to stop talking about Left/Right, Liberal/Conservative for instance).  And it’s certainly been a great tool for Gartner.  It gets them a lot of attention every year, like clockwork.

But the more I use PowerPivot the more I realize that the magic quadrant is too limited to handle the changes we’re experiencing in the world of data.

Gartner Measures Product and Vendor Capabilities

Look at the axes on Gartner’s quadrant.  They are “completeness of vision” and “ability to execute.”  I know a lot of detail goes into each axis but essentially each axis measures feature set provided by the vendor – current features, ability to add the right kinds of future features, ability to support their product, ability to remain a viable business, etc.

Those are valuable things to measure but they are not the only things that are important.

How About ROI?

In essence, the Gartner quadrant is a 2d checklist/scorecard of features (both of the product and the vendor).  Wouldn’t it be just as valuable to measure the average effectiveness and ROI as experienced by the customers of those vendors?  I’m sure Gartner factors that in to an extent of course, but I want to go farther:

I want ROI/Effectiveness to BE the axes on which we are measuring.  That’s the only thing that matters, in the end.  How effective is it for you, and how much did it cost you?

And let’s face facts:  satisfaction rates with BI tools and projects are almost universally low.  So when you start down the road of evaluating BI tools, you are often trying to decide which one is going to “suck the least” for you.

How About Tools That Aren’t Considered Traditional BI?

There are a lot of tools out there to help you with your data.  Not all of them fall under the heading of traditional BI tools.

Every day, organizations everywhere ask themselves not only the question “which BI tool for me?” but they also ask themselves “should we even try to get a BI tool in play here?”  (Oftentimes they are even implicitly asking themselves that question, thousands of times a day, and implicitly answering with “no, we’ll stick with spreadsheets.”)

I get asked, all the time, how PowerPivot compares to normal Excel.  How it compares to traditional BI tool X.  How it compares to data mining.  How it compares to custom-coded reporting solutions.

Invariably my answer includes a LOT more than feature set.  I always come back to ROI.

Example:  Building Materials Distributors

We’re currently writing a case study with Microsoft about one of our customers, Building Materials Distributors.  I won’t share the whole thing here until it’s done, but here are some relevant highlights:

1. They essentially had no BI capability two years ago.
2. They  purchased a number of solutions and none of them worked out.
3. They evaluated Cognos, but simultaneously were intrigued by PowerPivot.  The management team there thinks very critically and independently, with a ‘rubber meets the road’ mentality.  So they decided to give us (and PowerPivot) a chance first.
4. They got the results they needed, from Pivotstream/PowerPivot, at a FRACTION of the cost of adopting Cognos.
5. They did NOT hire any new staff to do this, and IT resources that would have been burned on BI were able to take on other critical projects.
6. It wasn’t just better in terms of price though:  they ended up getting BETTER INSIGHTS from Pivotstream/PowerPivot than the goals they had set for Cognos.  This was not something they expected from the “cheaper” platform.

Shouldn’t we be talking about this sort of thing?  And by the way, this isn’t rare.  I see this everywhere I go.  BMD happens to be a textbook case of it, but FAR from the only case.

Questions for YOU

First off, this is a beta

So I want your feedback:

1. What should I name the four quadrants?
2. I debated adding color to indicate feature completeness/maturity, what do you think?  (I’d give PowerPivot a mid-range color – strong and mature thanks to Excel and SSAS’s rich histories, but younger in terms of other BI features like Mobile, number of web controls, embeddability, etc.)
3. I decided against putting QlikView/TIBCO/etc. on the quad this time because I don’t know those tools very well and it would be hard for me to avoid some bias I think, but feel free to weigh in.
4. What other tools should I include?
5. Just general feedback/comments welcome too of course.

Leave me a comment, or email me.  I am Rob.  At a place called Pivotstream.  Dot com.

Got a question on the forums the other day.  It took some extra twists and turns but the simplest version of the question is worth covering here:  how do I perform a weighted average?

Let’s say you have some data about ZIP (postal) codes in the United States:

And you build a simple pivot that shows total population and median age for each ZIP, grouped by that last column, which is how fast the population of that ZIP is growing:

Simple SUM Measure for Population, Simple AVERAGE for Median Age

But that “Average Median Age” measure is just:

[AVERAGE MEDIAN AGE]=
   AVERAGE(Zips[Med Age])

which treats all ZIP codes as equal, even if they have wildly different populations:

The Median Age of Very Low-Population ZIP Codes (Like Pop=277)
is Counting Equally to Higher-Pop ZIP Codes (Like Pop=21484)

Correcting so that Median Age is Weighted by Population

Let’s write a new measure:

[Average Med Age Weighted by Pop]=
SUMX(Zips, Zips[TTL Pop] * Zips[Med Age]) / [Total Population]

And then view the results:

PowerPivot Weighted Average Measure Compared to Non-Weighted Average

And that is what we’d expect – since the smaller ZIP codes of population 277 and 427 both have very young median ages, they were “unfairly” skewing the subtotal younger.  When overall, fast-growing ZIPs have a lower median age once you weight them for population.

Even cooler – I can remove the ZIP code field from the pivot and the weighted average measure still works:

Weighted Average Measure Works Even When
the “Child” Field (ZIPCode) is not on the Pivot

For more information on SUMX(), see this post.

Actual Web Browser Screenshots of PowerPivot Sites:
Example of Homepage (top) and Report Page (bottom)

Want to see (and share) your workbooks on the web?

For a few months now we’ve been running free 30-day free trials where you can:

1. Upload workbooks to the Pivotstream cloud
2. Interact with them in the browser
3. Securely share them with colleagues, even if they don’t have PowerPivot or Excel 2010 installed

But we’ve been doing so in “low-profile” mode.  Meaning, if you contacted us, you found out about the trial.

I’m pretty stoked

I’ve been itching to announce the free trials so that you folks can try it out for quite some time.  But I’ve been held up on two things.  One is that we needed a revised and updated version of our brochure/tech specs document.  It’s done now and linked below.

The other obstacle was a debate over the perceived need to “qualify” people as business opportunities before providing a trial.  I’ve been pushing for a much more open process and eventually won people over.  If you try it out now and decide that you don’t need something like this yet, no problem.  I’m happy to expose people to the benefits of the web publishing experience, period.  And I know that later on, when your org is ready, you’ll likely remember us.

Besides, this stuff is just fun

Caveat:  No PowerPivot V2 Workbooks Until Next Month

Because we operate a service for thousands of users, we have to be very careful about rolling out new software on the servers.  There are a million moving parts in a software project like PowerPivot 2012 and we can’t just flip the switch – we need to validate many production scenarios in a test environment first.

That whole validation process is underway right now, but it takes time to do it right.  Sometime next month, we will be able to offer trial sites where your V2 workbooks function properly.  For now, all workbooks need to be produced in V1.

So if you’d like to sit tight and wait for V2 support on the server, I get that.  I will make another announcement here when it’s ready.

Workbooks Built in Version 10 (V1) Work on Our Servers Today
Workbooks Built in Version 11 (V2) – Wait for Next Month

Getting Started

Download brochure from here for more info on the full solution (not just the trial).

View some live demos here, on a live cloud PowerPivot site.

Email trials@pivotstream.com to get started.