On Break for Holidays. Next Post January 2nd!

December 20, 2011

 
image

Back in January with Renewed Vigor and Two Posts Per Week!

Well folks it’s been 15 straight weeks of two posts per week (Tuesday and Thursday), sometimes with bonus posts.  Your PowerPivot bloggers are taking a bit of a break over the next two weeks and then resuming our twice-weekly program with a fresh and rested perspective.

In the meantime we may have a bonus post or two, but no guarantees until January 2nd Smile

Have a great holiday season everyone and we’ll see you in two weeks.


Cash Flow Statement in PowerPivot

December 15, 2011

 

Guest Post by David Churchward

Cash Flow Statement

There’s few, if any, reports that I consider to be more important than the Cash Flow Statement.  Every business needs one, probably above any other report.  There’s numerous different versions of Cash Flow reports and, in reality, it can become a very personal thing to a company so I’m not going to propose to provide a one size fits all report.  In addition, I’m not going to proclaim to provide an IAS7 – Cash Flow Statement (that’s International Accounting Standards) textbook report because this isn’t designed as an accountancy lesson (but, if you’re interested, the method I’ll show here relates to the Indirect Method in IAS7).

With that said, if you consider this post alongside Kasper’s excellent post Building a cash flow statement in PowerPivot using Dynamic Measures in DAX, you should have all the tools you need to produce any Cash Flow Statement that you require.

The Accountancy Bit (very quickly!)

Just before we start, a quick accountancy point to consider.  Kasper’s approach considers cash payment and receipt transactions.  My model uses all transactions in the general ledger with the exception of Bank transactions.  On the basis that double entry exists (ie every transaction has an equal and opposite effect), all transactions that aren’t the bank transactions must equate to the same value.  The reason we use the “other” side of the account is because this side of the transaction tells us what the cash transaction was for.  As a result, we should simply be able to aggregate transactions into Cash Flow categories.

This seems quite straight forward.  Unfortunately, there are a few exceptions that need to be handled.  I’m only going to explain one which should give you the means to adjust for others.

Depreciation is purely an accounting adjustment.  This has the effect of crediting the balance sheet and debiting the P&L.  The effect of crediting the balance sheet has the effect of reducing the value that will be shown as Fixed Asset additions.  We therefore net off the depreciation transaction as if it never happened and I’ll show this later in this post.

The Dataset

My dataset (FACT table) contains 3 columns:

  1. GL_Account– this is my general ledger code.  Every transaction that forms the trial balance will carry a general ledger code and this tells us what the transaction was for.
  2. Month_End_Date – this date tells us the financial period to which the transaction relates.  I would advise using the month end date of the financial period in which the transaction will be reported as I’m doing in this example. (or a period number as in Rob’s Greatest Formula In The World posts).  It should be noted that a transaction date may not be in the same calendar period as the reporting date so care should be taken.
  3. Amount– this is the transaction value.  Using normal accounting signage, this will be a positive value for debits and a negative value for credits.

My dataset contains all transactions processed on the system – this is PowerPivot, it can handle it – and this essentially forms the basis of the trial balance.  If you have a lot of old data and don’t want to include it all, you could substitute historic data with opening balance values for each general ledger code.  This simply aggregates a lot of old transactions into one balance (per general ledger code) for an opening balance date, whilst maintaining the overall integrity of a balanced trial balance and the associated values.

You may have a dataset that holds monthly values for each general ledger code as opposed to each individual transaction.  This will work equally well although you won’t have the means to dissect the Cash Flow Statement by subset records such as customer or project.

If your data is held as monthly columns, it’s worth running an UNPIVOT.  Whilst we could condition PowerPivot to work with individual values for each period, it’s more efficient to work with one value column with associated period dates held in another column.

Related Tables

I link my my FACT table to a GL_ACCOUNTS table on the GL_Account code.  The GL_Accounts table provides details relating to the GL_Code such as account name and  report heading codes.  These heading codes, in turn, link through to a further GL_Headings table which holds cash flow report heading codes.  This, in turn, links through to cash flow heading tables.  It’s these headings that we will use on our Cash Flow Statement report.

Cash Flow Statement Dataset Schema

image

You’ll notice that CF1 doesn’t link to any other tables.  I’ll come onto that shortly.

General Ledger Account Mappings

The GL_Headings mapping is the critical element to this dataset.  This provides all of the key relationships between a general ledger code and the Cash Flow Statement report heading that you want the account to be represented under.

GL_Headings Table

image

My GL_Accounts map through to my Heading_Code on this GL_Headings table.  This table holds all of my key report headings and provides a useful grouping of accounts.  For instance, all general ledger codes that relate to Sales will carry a Heading_Code of 1010.

These definitions are important.  This is a one-off setup and you’ve probably done this indirectly in some way or another to drive any reports or Excel analysis that you’re currently doing (probably with VLOOKUP).  This process simply affirms that mapping.  Ideally, you would code these heading codes into your General Ledger account code setup and formulate the code allocation as part of the process to set up new general ledger accounts.

You’ll recall that I mentioned Depreciation earlier in this post.  Whilst this is a P&L account (heading code 1045 in this example), I’m coding this to exactly the same place as Fixed Assets on the Balance Sheet (heading code 2010).  This effectively eliminates the deprecation transactions.  An alternative would be to filter out the accounts in question.  You can do this, but be careful to ensure that those accounts net to zero.

Each of these codes map through to Cash Flow Statement heading tables.  These heading tables can be whatever you need them to be, with headings and ordering assigned however you want your final report to be represented.  The one’s I’m using are shown below.

CF2 – Cash Flow Statement Heading 2

image

This is my first grouping of headings.  These align with the IAS7 headings for CF2 codes 1, 2 and 3 and basically split my Cash Flow Statement between Operations Activities, Investing Activities and Financing Activities.

There’s a fourth section called Bank.  The Bank general ledger codes map through to this code and we eliminate them for the Cash Flow element of this report.  However, we bring them back into action when we display the Bank Balances at the end of the report.

CF3 – Cash Flow Statement Heading 3

image

This is the next level of granularity on my report.  The Bank general ledger codes don’t have any CF3 mapping and yet I’m carrying Opening and Closing Bank Balance codes on this heading set.  I’ll come onto why that is a bit later.  You’ll also notice that I’ve coded these headings with an Open_Bank  and Close_Bank flag.  We’ll use dynamic DAX measures later on to utilise these.

Note – there should NOT be any mapping of accounts to the CF3 codes 10 and 11 in this table (ie Open and Close Bank Balance).  We’ll use these simply to drive a DAX measure later.

CF4 – Cash Flow Statement Heading 4

image

One element of my Cash Flow Statement will carry a further breakdown – Working Capital (although you can have more if you want).  I’ve done this simply to provide an extra level of granularity, but you could decide to replace the Working Capital heading in CF3 with these headings and map accordingly.

I’ve got a few reasons for doing this:

  1. I like to show my Working Capital elements specifically – this targets the key operational balance sheet items that generally need to be tracked
  2. This helps to line up a future post!

CF1 – Cash Flow Statement Heading 1

You might, quite rightly, question why my ordering has gone to pot and I’m explaining CF1 last!  Quite simply, this table sits out on it’s own because it simply drives report sections as opposed to being a mapped heading code.  I’m going to show a Cash Flow section and then, below that, I’m going to show the effect on the bank balance.  No relationships required.

image

And Now for the DAX – Cash Flow

I’ve got to be honest, this measure isn’t going to set the world alight.  However, this goes to show how easy PowerPivot can be, especially if you condition your data properly.  I’m creating a measure aptly called Cash_Flow

Cash_Flow

=CALCULATE(

SUM(FACT[Amount]),

CF2[CF2_Name]<>”BANK”

       )*-1

This is a simple CALCULATE function which sums the [Amount] field on the FACT table with a simple filter set to ignore any GL_Accounts that have a CF2 category mapping of “BANK”.

You’ll notice that I multiply this by –1 to reverse the signage.  In simple terms this is because of the double entry effect.  We’re using the “other” side of the transaction which means that the sign convention is going to be back to front.  This simply reverses that effect.

When I pull this measure onto my report together with the CF1,2,3,4 heading names, I get the following.

Cash Flow in Excel - PowerPivot Turns Complex into Simple

You might say that this is job done (especially if you remove CF1 heading from the report). This does give us a Cash Flow Statement. However, this doesn’t give my bank balances and, if I include my CF1 heading, my report is repeating the same data for both of the CF1_Name headings. This is understandable as CF1 is not linked to my FACT table in anyway.  This is intentional (believe it or not!).

Bank Balances

I want to show opening and closing bank balances for the Bank section of my report and this is where it gets a bit more interesting.  Let’s create a measure that calculates these balances and let’s call this measure Bank…

Bank

=IF(COUNTROWS(VALUES(CF3[CF3]))=1,

IF(VALUES(CF3[Close_Bank])=1,

CALCULATE(

SUM(FACT[Amount]),

ALL(CF3),

CF2[CF2_Name]=”BANK”,

FILTER(

ALL(Month_End_Dates[MonthEndDate]),

Month_End_Dates[MonthEndDate]<=

MAX(Month_End_Dates[MonthEndDate])

            )

          ),

IF(VALUES(CF3[Open_Bank])=1,

CALCULATE(

SUM(FACT[Amount]),

ALL(CF3),

CF2[CF2_Name]=”BANK”,

FILTER(

ALL(Month_End_Dates[MonthEndDate]),

Month_End_Dates[MonthEndDate]<=

EOMONTH(MAX(Month_End_Dates[MonthEndDate]),-1)

  )

),

BLANK()

   )

    )

      )

We should be offering prizes for those of you would can instantly spot the GFITW elements!!  Anyone would think that this had somehow been contrived!  Believe me – it hasn’t.  This is a concept that I use regularly and it’s a joy to see Rob explain it so well so I don’t have to!

So what is this doing?  I’ll explain as clearly as I can, but it’s worth paying very close attention to Rob’s Greatest Formula In The World posts.  I won’t re-iterate too much here.

  1. The first IF statement simply serves to provide an answer as to where I want the measure to be evaluated.  I want an answer when my CF3_Heading is flagged as Open_Bank or Close_Bank.
  2. The second IF statement creates a value when my CF3_Heading [Close_Bank] is flagged.  Where this is the case, the following happens:
    1. Sum the [Amount] Field
    2. Expand my dataset to all CF3 headings in order to release the dependency on the current CF3 heading being evaluated – ALL(CF3)
    3. Apply a filter to include all of my Bank general ledger codes – CF2[CF3_Name]=”BANK”
    4. Apply a further filter (FILTER) that:
      1. Removes the dependency on the MonthEndDate – ALL(Month_End_Dates[MonthEndDate])
      2. Apply a filter that includes all months that pre-date and include the reporting month – Month_End_Dates[MonthEndDate]<=MAX(Month_End_Dates[MonthEndDate]) – this is important because the transaction base for the month won’t give us the balance.  We need to include ALL transactions that pre-date and include the reporting date.
  3. The third IF statement has a similar effect to the second IF statement, but it produces an opening balance and therefore the current reporting month has to be ignored.  As a result, I want to sum all transactions where the date is less than or equal to my prior month date – Month_End_Dates[MonthEndDate]<=EOMONTH(MAX(Month_End_Dates[MonthEndDate]) ,-1)

Cash Flow in Excel with Bank Balances Thanks to PowerPivot

Once again, we have the correct results but we’re getting values repeated for both CF1 headings.

Sorting Those CF1 Headings

We now have our two base measures.  All we now need to do is evaluate when to use each one.  Let’s create another measure called Cash_Flow_Statement.

Cash_Flow_Statement

=IF(COUNTROWS(VALUES(CF1[CF1]))=1,

IF(VALUES(CF1[CF1])=1,

[Cash_Flow],

[Bank]

   )

)

This measure simply evaluates for CF1 headings and determines that if my CF1 Heading Code is 1 (ie Cash Flow) then use my [Cash_Flow] measure.  Otherwise, use my [BANK] measure.

Cash Flow Finished

A bit of formatting and there you have it.  You’re just about done.  Or rather, you’re done if you think that’s enough.  I, however, don’t like some of these totals and duplication of values in certain areas.  There’s a way to deal with those totals but that’s a topic for a future post.

So what else can we do with this?

Just as a taster, here’s something that I like to call a Cash Flow Bridge.  This could be a topic for a future post, but just to whet your appetite……

Cash Flow Statement Bridge

If there’s enough call for this, I’ll get this scheduled in for a future post.  Neat eh!


The Greatest Formula in the World, Part 3

December 13, 2011

 
Our time machine that we developed in part one and part two still needs some explanation, so let’s dive in.

First Things First:  The MAXX Wasn’t Needed

Looking back, I noticed that I was overcomplicating things needlessly.  The final formula I had involved the following FILTER() term, in which I used a MAXX:

FILTER(ALL(Periods),
       Periods[NextYear Period]<=
          MAXX(VALUES(Sales[Period Num]), Sales[Period Num])
)

Turns out that MAX() works just fine, so let’s replace that clause and simplify things a bit.  Here’s the new formula:

CALCULATE([Total Sales],
   ALL(Periods),
   FILTER(ALL(Periods),
      Periods[Year]=MAX(Periods[Year])-1
   ),
   FILTER(ALL(Periods),
          Periods[NextYear Period]<=MAX(Sales[Period Num])
   ),       
   VALUES(Periods[MerchPeriod])
)

OK, with that, we can move on to explanation:  How does this formula work??

I used to call this technique “expand then filter”

Well actually I still do, in my own head.  It’s just that the GFITW is a catchier title.

OK, so the “expand” part is just that first ALL():

CALCULATE([Total Sales],
          ALL(Periods),

The first step in the formula, then, is basically just telling the calc engine to throw away all filters on the Periods table.  In other words, “forget all concept of time, pretend the pivot is not filtered at all with respect to time.”

We do this so that we have a clean slate.  Then, in the subsequent FILTER clauses, we build up a new filter context for time.

One more time for clarity:  the way this formula works is to first throw out all time filters, and then in subsequent steps, we build up new filters to match the time period that we want, which in this case is last year.

Once you understand that, this overall formula starts to get pretty simple.  Each piece of the formula is quite straightforward in its own right.  ALL() is pretty straightforward for sure, and so are the subsequent FILTERS().

OK, we’ve expanded.  Now on to the filters!

The first filter says, “hey, now that we’ve thrown out all time filters, let’s filter time back down to just be last year.”

FILTER(ALL(Periods),
       Periods[Year]=MAX(Periods[Year])-1
)

Let’s talk about the FILTER() function itself for a moment.

How does FILTER() Work?

Honestly this function has deserved its own post for a long time.  I’ll give a brief explanation here.

The syntax for the FILTER function is FILTER(TableToFilter, FilterExpression).   Pretty simple.  Here’s some more detail:

  1. FILTER() takes a TableToFilter and a FilterExpression, and returns all rows from that TableToFilter that match the FilterExpression.
    1. In the example above, TableToFilter is ALL(Periods)
    2. and FilterExpression is Periods[Year]=MAX(Periods[Year])-1
  2. FILTER() steps through the TableToFilter one row at a time. 
    1. And for each row, it evaluates the FilterExpression.  If the expression evaluates to true, the row is “kept.”  If not, it is filtered out.
    2. Because FILTER() goes one row at a time, it can be quite slow if you use it against a large table.  When I say “large” that is of course subjective.  A few thousand rows is fine in my experience.  A million is not.  Do not use FILTER() against your fact table.
  3. The FilterExpression typically takes the form of Table[Column] = <expression>
    1. The comparison operator doesn’t have to be “=”.  It can also be <, >, <=, >=, <>
    2. The expression on the right hand side of FilterExpression can be “rich.”  This is VERY useful.  In a simple CALCULATE, the right side of each filter expression has to be simple, like a literal number (9) or a string (“Standard”).  The fact that FILTER() allows for rich expressions here is one of the most common reasons I use FILTER().
    3. The Table[Column] in the filter expression is a column in the TableToFilter.  If you are filtering the Periods table, it makes sense that you are testing some property of each row in Periods.  I can’t think of a sensible reason to use a column here that is NOT from TableToFilter.  (Insert “boot signal” here, maybe the Italians can address this).
  4. FILTER() ignores everything else going on in your formula and acts completely on its own.
    1. For example, our overall formula sets ALL(Periods) as the first argument to CALCULATE.
    2. The FILTER()’s that come after that do NOT pay any attention to other arguments however, including that ALL(Periods).
    3. In other words, the FILTER() functions are still operating against the original filter context from the pivot!  If the pivot is sliced to Year=2009, then the FILTER() function starts with the Periods table already pre-filtered to just 2009.
    4. This is why each of my FILTER()’s uses ALL(Periods) for TableToFilter.  I have to repeat the “expand” step so that my FILTER() is also working from a clean slate.
  5. Even though each FILTER() operates on its own, their results then “stack up” in the overall formula.
    1. Even though FILTER() RETURNS a set of rows that matched the FilterExpression, it actually REMOVES rows from the overall filter context.
    2. This sounds tricky but really it isn’t.
    3. Let’s say our TableToFilter contains 6 rows:  A, B, C, D, E, and F.
    4. And our overall formula contains two FILTER() clauses that both operate on the same TableToFilter, just like our overall formula near the beginning of this post.
    5. Let’s also say that the first FILTER() returns rows A, B, C, and D.
    6. And the second FILTER() returns rows C, D, E, and F.
    7. The net result is that only rows C and D are left “alive” in the overall filter context of the formula.
    8. So one way to think of this is that FILTER()s “stack up” on top of each other.
    9. Another way to think of it is that even though the first filter RETURNED rows A, B, C, and D, its real effect was to REMOVE all other rows (E and F) from consideration.

OK, back to that first filter!

Here it is again:

FILTER(ALL(Periods),
       Periods[Year]=MAX(Periods[Year])-1
)

Let’s revisit points 1-5 above for this FILTER expression to see how it all works.  And let’s examine just a single cell of the pivot to see how this FILTER operates for that one cell:

image

Focusing on The Orange-Circled LASTYRSALES Cell As An Example
With Its Period Filter Context Highlighted in Green

In that picture above, the orange cell we are looking at has a filter context “coming in” from the pivot.  It has Period[Year] set to 2011 and Period[MerchPeriod] set to 1, as highlighted in green.

Given the detailed description of FILTER() from points 1-5 above, we can see that:

  1. We set ALL(Periods) as our TableToFilter so that we are starting from a clean slate with respect to time.  So our Periods table now has “all rows alive.”
  2. Then our FilterExpression tests against the Periods[Year] column.
  3. MAX(Periods[Year]) – 1 still operates independently!, so it still picks up Periods[Year]=2011 from the pivot.  Therefore it returns 2011 – 1 = 2010!
  4. Since we started with ALL(Periods) as the TableToFilter, and the FilterExpression only “keeps” rows where Year=2010, we are left with all 2010 rows “alive” after evaluating this FILTER().
  5. If we didn’t do ALL(Periods) for TableToFilter, and instead just used Periods without the ALL(), our FILTER would start out with only rows from 2011 (since that is what the pivot is telling us).
    1. And then in the next step when we go back a year to 2010, FILTER() would find no rows.  There are no rows that match Periods[Year] = 2010 and Periods[Year]=2011. 
    2. So our FILTER would return no rows, which means it would have the effect of REMOVING all rows from Periods in the overall formula, and our measure would return blank for all cells in the pivot.

That’s a lot of explanation, I know.  Walk through it a few times.  It’s actually pretty intuitive once you’ve done it a few times.  The tricky part, for me, was discovering all of these details for myself.  And since I’ve done all of that, you don’t have to.

That’s enough for this time.  I think you can probably figure out how the second FILTER() evaluates based on the above, but I will step through it next time. 

I will also explain why we use that last VALUES() in the formula, and probably also share some of the answers I got from the Italians, and from David Churchward, in response to my question “did I need to add that calc column in the Periods table?”


Friday Bonus: Gang Sign Photos So Far

December 8, 2011

 

This is my favorite so far, by the narrowest of margins.  It comes from Angel Abundez of RDP Streamline Solutions.  And Angel (twitter) executes quite well.  Note the form.  Good X – not the V or the Y that most people’s X’s are turning out to be.  And a sharp L.  The background pic is nice too.

But his daughter simply nails it.  Look at that total lack of emotion.  A stone cold spreadsheet killa.  Everyone should aspire to her standard of excellence.

VIEW THE FULL GALLERY HERE

I’m gonna leave submission open for a few more days.  Plenty of time to get yours in!

SUBMIT YOUR PICTURE(S) HERE


The Greatest Formula in the World, Part Two

December 7, 2011

image

Our Time Machine From Part 1 Needs a Couple of Fixes

OK, if you recall from part one, we had a “year over year” time machine calculation going on, and it was built against a data set that lacked a “real” calendar table.  Very cool.  Very resourceful.  And very necessary.

But the time machine had two problems.  Let’s fix them.

Problem #1:  Meaningless Grand Total

Let’s start with the easy one:

Year over year in PowerPivot - Meaningless Grand Total

Grand Total is Meaningless for Last Year Sales

It is meaningless to have a “grand total” value for a measure that returns “last year’s sales.”  What year would that BE, actually?  It’s nonsense.  So we use an old trick, one of the many flavors of IF(VALUES()):

=IF(COUNTROWS(VALUES(Periods[Year]))>1,BLANK(),
   CALCULATE([Total Sales],
      ALL(Periods),
      FILTER(ALL(Periods), Periods[Year]=MAX(Periods[Year])-1),
      VALUES(Periods[MerchPeriod])
   )
)

Where the new IF “wrapper” is highlighted and the original formula is in normal font.

For more on that “IF VALUES” technique, please see this post.

The results are as desired, the grand total is now blank:

image

Problem solved.  Moving on…

Problem #2:  2011 isn’t complete but we’re getting all of 2010 sales

Remember, our sales data for 2011 only goes through the first 6 months.  So we do NOT want “last year sales” for months 7-12:

image

That can be solved a number of ways.  The simplest is just to add another IF(), and have the Last Year Sales measure return BLANK() whenever there are no Sales:

=IF(COUNTROWS(VALUES(Periods[Year]))>1,BLANK(),
   IF([Total Sales] = BLANK(), BLANK(),
      CALCULATE([Total Sales],
         ALL(Periods),
         FILTER(ALL(Periods),
            Periods[Year]=MAX(Periods[Year])-1
         ),
         VALUES(Periods[MerchPeriod])
      )
  )
)

That trims the results to just the months desired:

image

But note that the subtotal for 2011 is still too high:

image

“Last Year Sales” for 2011 Should Only be Returning
2010 Sales Through the First Six Months

This one is trickier.  I struggled to find a good answer before coming up with the following:

Step 1:  Add a “Next Year Period Num” Column to the Periods Table

Recall that my Sales table has a “Period Num” column that is the basis for the relationship with the Periods table:

image

And that matches up with a similar column in the Periods table:

image

Note that Period Num does NOT reset to 1 with each new year (unlike MerchPeriod).  So that is the absolute unique ID for a given Month/Year combo.

Well, I created a new column in the Periods table that tells me, for a given Period Num, what the equivalent Period Num will be NEXT year:

image

Now, in my FILTER statements, I can choose to match on THAT column instead.

Bringing it Home

OK, now that I have that column, I can add a new FILTER clause to my measure:

=IF(COUNTROWS(VALUES(Periods[Year]))>1,BLANK(),
   CALCULATE([Total Sales],
      ALL(Periods),
      FILTER(ALL(Periods),
         Periods[Year]=MAX(Periods[Year])-1
      ),
      FILTER(ALL(Periods),
         Periods[NextYear Period]<=
            MAXX(
                VALUES(PeriodSales[Period Num]),
                PeriodSales[Period Num]
                )
      ),
       
      VALUES(Periods[MerchPeriod])
   )
)

OK, what does that do?  Simply put, it further filters the Periods table to NOT go beyond any periods for which we currently have records in the Sales table.

And the results:

Last Year Sales Measure in PowerPivot With a Custom Calendar

Last Year Sales Measure Doing the Right Thing

And now if you want YOY Growth, it’s straightforward:

=([Total Sales] – [LY Sales Finished]) / [Total Sales]

Which yields:

Year over Year / Year on Year Growth Percentage Measure in PowerPivot With a Custom Calendar

Year over Year / Year on Year Growth Percentage
Measure in PowerPivot With a Custom Calendar

OK, I cheated and wrapped another IF(COUNTROWS(VALUES)) around the outside of the formula so that the grand total cell is blank again, because again, that’s a meaningless value to report.  Details.

Isn’t that a lot of work?

Depends on how you look at it.  First of all, it’s a lot harder to grasp the first time than it is as you get used to it, trust me.

But even better, this investment pays off forever.  Normal Excel formulas may be easier to write the first time, but then you have to re-write and adjust them forever, every time your data changes.  And every time your desired report shape changes.

This one is a portable formula.  It goes wherever you want it to go, and eats whatever data you feed it.  Forever.  It’s worth a little extra effort.

OK but do I really have to understand all of that?

No, not really.  Not right away.  Think of it this way:  this is what I do all day, every day.  And I can tell you that figuring this out is not something you want to do in your spare time.

But once someone gives you the pattern?  Wash, rinse, repeat.  Make small adjustments as needed.

Honestly, that whole formula above could be wrapped up in a function provided by Microsoft, and all of the nasty stuff hidden.  I told them as much last time I visited Redmond.  And if it were a function rather than a formula, we wouldn’t care at all how it worked.

So… treat this as a pattern.  Copy/paste and modify to fit your needs.  Seriously.

Unfinished Business

I actually WILL retrace my steps here though and explain a number of things though, so that you CAN understand.  I have a couple of posts in the queue aimed at just that.

Were there alternate ways to write this measure?

Yes, there were, especially in that last step where I needed to get the “Last year sales” measure correct for the 2011 subtotal.  I could have used SUMX to make the year subtotal equal to the sum of its underlying months.  That is worthy of a post for sure.

But I was wondering if there was still another way.  A way that doesn’t use SUMX and doesn’t require a new calc column.

And for that, we need the Italians.  So I’m gonna try out my new toy, the Boot Signal:

image

Calling Marco and Alberto

Get it?  *Boot* Signal?


The Greatest Formula in the World, Part One

December 6, 2011

 
image

Play Write the greatest song formula in the world, or I’ll eat your souls.”
-From “Tribute,” by Tenacious DAX

A Do-It-Yourself Time Machine

You remember the Great Precedence Project?  Well, I have shelved it for now, for two reasons:

  1. Creating an exhaustive list of all the precedence rules in PowerPivot formulas was going to be…  exhausting.
  2. I didn’t really need to boil the philosophical ocean in search of deep underlying meaning, because on further inspection, I already had the Greatest Formula in the World, and needed no other.

So what is the GFITW about?  It’s about time navigation.  It’s useful when you have a custom calendar and the time intelligence functions are therefore not so helpful.

The GFITW is a do-it-yourself time machine.

Year on Year Sales With a Custom Calendar?  Yes We Can!

In that post linked above, which happened to be the last installment of the precedence project, I was calculating a running total, and doing so without benefit of the fancier functions like DATESYTD etc. (which rely on the “real” calendar, not your business calendar).

OK, so now let’s do a “Year on Year” sales measure, sometimes called a “Year over Year” measure.  And again, our “calendar” table isn’t a true calendar at all:

Semantically this IS a calendar but functions like PREVIOUSYEAR can't use it

A Table of Periods (Not Dates) Means You Cannot Use the Built-In
Time Intelligence Functions to Calculate Year-on-Year Sales

And the Sales table is very simple:

Sales Table linked to a period number, not linked to a date

Sales Table is Linked to the Periods Table by the Period Num Column

And I have this basic pivot already set up:

image

Year and MerchPeriod on Rows

So, how do I write a measure that “fetches” the sales from last year?

Let’s Skip Some Steps

I am going to be honest with you:  it took me a very long time to figure this out.  The formula looks reasonably simple, and it is, but I had to try a million variations before I got it right.  Literally, it took me about a week to refine it.

So rather than walk you through that painful process, let’s skip ahead and show you the GFITW in all its glory.  Then in the next post I can show you what NOT to do, and we can also fine tune it.  OK?  OK.

So here it is, the Greatest Formula.  In the World:

=CALCULATE([Total Sales],
   ALL(Periods),
   FILTER(ALL(Periods),
      Periods[Year]=MAX(Periods[Year])-1),
   VALUES(Periods[MerchPeriod])
)

And its results:

image

No, it’s not perfect yet.  We will need to calculate percentage growth of course, rather than merely fetching last year’s sales.  And that given that 2011 is not yet complete, I’m a little squeamish about Last Year Sales returning ALL of 2010’s sales (the $5.9M number).

But for now, let’s count our blessings.  This formula DOES fetch the sales from last year, and it works (mostly) both at the year level and the MerchPeriod (aka Month) level.

Dissecting the Formula

Here is the formula again:

=CALCULATE([Total Sales],
   ALL(Periods),
   FILTER(ALL(Periods),
      Periods[Year]=MAX(Periods[Year])-1),
   VALUES(Periods[MerchPeriod])
)

But really, the GFITW is not a formula.  It is a pattern that you can re-use and modify.  So let’s look at it that way:

=CALCULATE(Original Measure,

   ALL(Your Custom Calendar Table),
  
   FILTER(ALL(Your Custom Calendar Table),
      Year Column Or Similar =
        
Expression that “moves” the Year Column
  
),
  
   VALUES(
Time Column That is More Granular
          Than Year, Like Month or MerchPeriod

  
)

)

Where everything in blue is “fixed” as part of the pattern, and everything in italics is something you can change.

Like I said, this is running a bit long for one post.  Come back Thursday for the rest Smile


Friday Bonus: Excel Goes Gangsta!

December 2, 2011

 
image

“You Better Step Outa My Grid”
Dick Moffat Helps Me Debut the “Official” Excel (XL) Gang Sign

I Think We Can All Agree This Was Long Overdue

See a need, fill a need.  That’s kinda my credo.  And we clearly needed a gang sign that represents the worldwide cartel of Excel Ninjas.  (And their innocuous front group, the Excel Monkeys).

OK maybe that wasn’t it.  Maybe I’ve just felt left out of the whole long-running fad of posting ridiculous pictures of yourself on Facebook, flashing gang signs (a practice that IMO should rank much higher than #6 on the list of 15 dumbest Facebook picture types, but it does get properly skewered in this music video).

OK that wasn’t it either.  Really, I just think this picture, and this IDEA, couldn’t POSSIBLY be any funnier.

An Opportunity for Ridiculous Irony

That fine example of American leadership, Rahm Emanuel, is purported to have said “never waste a crisis.”  Similarly, I like to say, “never waste a visit from Dick Moffat.”  We recently met up to swap DAX techniques and war stories, and I knew we had to do a picture.

“What would be most ridiculous?” was naturally my first thought, and what emerged was, in my opinion, some of my finest work:  the XL gang sign.

I mean, who could be a better cohort for this than a pacifist Canadian grandfather, Access/Excel MVP, and author of a top-5 all-time popular post on… wait for it…  cube formulas?  (That post is 1.5 years old and STILL gets 50 views a day!)

I mean, the most “gangsta” Dick has ever looked was back when he used to do oatmeal ads:

image

A Bold Move:  The Gang Sign Goes Open Source?

When discussing this with our COO (Frank Fulton), he pointed out that this was not something we could responsibly exploit for our own benefit, and that we were obligated to open source it.

It took some time for me to warm to the idea.  I mean, while still basking in the glow of invention, it’s hard to change gears and immediately just give it all away like that.  I mean, at least let me protect it with the GNU Public License or something.

But I think it’s for the best.  Precise licensing terms will be announced shortly, once our massive legal team has had time to iron out a few issues with the Federal Court of Appeals.

Show Dick How it is Done – Submit Your XL Gang Sign Pics!

If you look again at that picture up top, it’s clear that one of us took to this task a little more naturally than the other.  I need your help.

Send me your pics of you (and maybe your spreadsheet crew) flashing the sign.  I’ll post the best ones next week and maybe even throw in a few small prizes.

Send pics to:

outamygrid@pivotstream.com 

Some suggestions:

  1. Look “hard.”  Sell it.  Scowl.  Look unfriendly but don’t overdo it and get snarly.  You’re not 1980’s Hulk Hogan.  You gotta have that cool indifferent look of badness, as modeled here by a more recent Hulk Hogan.
  2. Get Creative with Your Irony. Standing in front of a bank of huge monitors on Wall Street would work.  A massive cubicle farm, also good.  Wearing an Apple t-shirt, whatever – just give it something.  I mean, I accessorized with Dick Moffat.  That sets a high bar right there folks.
  3. Pay attention to details of the sign itself.  Remember, you have to make it backwards for you so that others see it the right way.  Left hand is the “L.”  Thumb points out (to your left).  Right hand is the X.  Index finger is closest to you and points to your right.  Middle finger crosses it to your left.  Or take advantage of the open source licensing and modify it to your liking.
  4. Channel your long-simmering feelings of disrespect.  You are the exploited, ignored backbone of the international business world.  Seriously, you are.  The entire “BI” industry claims to do what is 99.99% done by YOU all day, every day.  Channel that rage.  Oh yeah, you know where this is going…
  5. Now is your time!  Remember, PowerPivot truly is a revolution.  Our revolution.  YOUR revolution.  Anyone who tells you that Excel will never “be real BI” or tries to pigeonhole it as “BI for small teams only” doesn’t yet understand how much PowerPivot has changed the rules.  Put THAT thought in your head… 

…and then go take a ridiculously funny picture.


Using Query Editor to Create a Time Table

December 1, 2011

 

Guest Post by David Churchward with Technical Input from Scott Senkeresty

The first thing I ever do when I create a new PowerPivot workbook is create the date tables that I need.  Until now, I’ve used a preconditioned SQL table to draw this information from.  Now, thanks to Scott Senkeresty, you can now create your own – from PowerPivot!

Scott is a true genius.  He spends a lot of time under the hood of Pivotstream operations working his magic and wizardry and, every now and then, he pops up for air to deliver a masterpiece like this.

 

It’s as Simple as Copy and Paste

If you read my last post SQL UNPIVOT Makes My Data Skinny, you’ll know where query editor is and the sort of tricks that it opens up within PowerPivot.  In this post, there’s some SQL scripts that create time tables and you can simply cut and paste them into the query editor and use them straight away.

Having said this, it’s worth checking out Rob’s post The Ultimate Date Table to see why a date table is important and where you can get one of your own from Azure DataMarket.  However, I think we’ve gone one step better now and given you a script that you can parameterise easily, condition relatively easily and setup extremely easily.  Add to that the fact that it runs in mere seconds!

AND there’s one more bonus, especially for Accountants, you can set this up for YOUR Fiscal Year!

Copy, Paste, Save – Simples!

To get going, simply copy and paste the script from THIS ATTACHMENT into query editor.  When you save it – bam – time table complete.

DECLARE @FirstDate DATE

DECLARE @LastDate DATE

SET @FirstDate =’1 Jan 2008′– Enter first date of calendar as ‘dd mmm yyyy’

SET @LastDate =’31 Dec 2015′– Enter last date of calendar as ‘dd mmm yyyy’

 

IF @FirstDate ISNULLOR @FirstDate =

                BEGIN

                                SET @FirstDate =’1 Jan 2000′

                END

               

IF @LastDate ISNULLOR @LastDate =

                OR @LastDate < @FirstDate

                ORDATEDIFF(DAY, @FirstDate, @LastDate)> 10000

                BEGIN

                                SET @LastDate =DATEADD(DAY, 10000, @FirstDate)

                END

 

                ;WITH CTE_DatesTable

                AS

                (

                  SELECT @FirstDate AS [Date]

         UNIONALL

         SELECTDATEADD(DAY, 1, [Date])

         FROM CTE_DatesTable

         WHEREDATEADD(DAY, 1, [Date])<= @LastDate

                )

 

                SELECT

                                Date,

                                DATEPART(DAYOFYEAR,Date)                             ASDayOfYear,

                                DATEPART(DAY,Date)                                          ASDay,

                                DATEPART(MONTH,Date)                                    ASMonth,

                                DATEPART(YEAR,Date)                                        ASYear,

                                DATENAME(WEEKDAY,Date)                               ASDayOfWeek,

                                LEFT(DATENAME(WEEKDAY,Date),3)                 AS DayOfWeekAbr,

                                DATEPART(WEEK,Date)                                       AS WeekOfYear,

                                DATEPART(QUARTER,Date)                                 ASQuarter,

                                DATENAME(MONTH,Date)                                  ASMonthName,

                                LEFT(DATENAME(MM,Date),3)                           AS MonthNameAbr,

                                DATEADD(DAY,-1,DATEADD(MONTH, 1,DATEADD(DAY,(DATEPART(DAY,

DATE)*-1)+1,Date)))                         AS MonthEndDate

 

                FROM

                CTE_DatesTable

 

                OPTION (MAXRECURSION 10000)

Without doing anything to this script, you’ll get a time table that spans Jan 2008 to Dec 2015.  That could be enough.  In which case, you’re done.  However, I’m sure you want more.

Change Date Parameters to Widen the Timeframe

The great news with this script is that it’s so easy to adjust to cover the timeframe that you need.  You just need to follow these simple couple of steps:

  1. Towards the top of the script you’ll see “SET @FirstDate = “.  You simply need to change the value to the first date in your dataset (or perhaps an earlier date to be sure).
  2. I’m sure you’ve already guessed step 2 – change “SET @LastDate = “ to a value that is beyond the last date in your dataset.

I almost wish I could make it more difficult but Scott’s nailed this so well that it’s almost impossible to get wrong – well almost!

A Quick Health Warning

This script uses a recursive CTE (Common Table Expression).  If you’re not careful, it’s very possible to spin this baby into an infinite cycle.  Without going into why this happens, here’s a couple of points to make sure that you don’t fall into that trap:

  1. The IF – BEGIN – SET – END statements are there to ensure that valid entries are made for @FirstDate and @LastDate.  It’s probably best to leave these in.
  2. At the end of the script, there’s a command OPTION(MAXRECURSION 10000).  This ensures that the script won’t loop more than 10,000 times (ie 10,000 days).  If you set the recursion to zero, this removes all limits and then you could get into an infinite cycle.

Don’t be scared to give it a go.  There’s controls here to stop anything going wrong and what’s the worst that could happen anyway!

The Next Level – Fiscal Elements

In all my time using SQL and PowerPivot, I’m yet to work in an environment where a year runs from 1st January to 31st December.  As a result, we refer to a Fiscal Year to represent this alternate calendar.  I took the liberty of hacking Scott’s script to add these elements in.

DECLARE @FirstDate DATE

DECLARE @LastDate DATE

DECLARE @FiscalStart INT

SET @FirstDate =’1 Jan 2008′– Enter first date of calendar as ‘dd mmm yyyy’

SET @LastDate =’31 Dec 2015′– Enter last date of calendar as ‘dd mmm yyyy’

SET @FiscalStart = 4 – First calendar period of Fical Year – 1=Jan, 2=Feb…., 12=Dec

 

IF @FirstDate ISNULLOR @FirstDate =

                BEGIN

                                SET @FirstDate =’1 Jan 2000′

                END

               

IF @LastDate ISNULLOR @LastDate =

                OR @LastDate < @FirstDate

                ORDATEDIFF(DAY, @FirstDate, @LastDate)> 10000

                BEGIN

                                SET @LastDate =DATEADD(DAY, 10000, @FirstDate)

                END

 

                ;WITH CTE_DatesTable

                AS

                (

                  SELECT @FirstDate AS [Date]

                 UNIONALL

                 SELECTDATEADD(DAY, 1, [Date])

                 FROM CTE_DatesTable

                 WHEREDATEADD(DAY, 1, [Date])<= @LastDate

                )

 

                SELECT

                                Date,

                                DATEPART(DAYOFYEAR,Date)                             ASDayOfYear,

                                DATEPART(DAY,Date)                                          ASDay,

                                DATEPART(MONTH,Date)                                    ASMonth,

                                DATEPART(YEAR,Date)                                        ASYear,

                                DATENAME(WEEKDAY,Date)                               ASDayOfWeek,

                                LEFT(DATENAME(WEEKDAY,Date),3)                 AS DayOfWeekAbr,

                                DATEPART(WEEK,Date)                                       AS WeekOfYear,

                                DATEPART(QUARTER,Date)                                 ASQuarter,

                                DATENAME(MONTH,Date)                                  ASMonthName,

                                LEFT(DATENAME(MM,Date),3)                           AS MonthNameAbr,

                                DATEADD(DAY,-1,DATEADD(MONTH, 1,DATEADD(DAY,(DATEPART(DAY,

DATE)*-1)+1,Date)))                         AS MonthEndDate,

                                DATEADD(DAY,-1,DATEADD(MONTH, 1 + 12 -(DATEPART(MONTH,DATE)

- @FiscalStart + 1 +CASEWHEN @FiscalStart >DATEPART(MONTH,DATE)

THEN 12 ELSE 0 END),DATEADD(DAY,(DATEPART(DAY,DATE)*-1)+1,

Date)))                                                   AS YearEndDate,

                                DATEPART(MONTH,DATE)- @FiscalStart + 1 +

                                                CASEWHEN @FiscalStart >DATEPART(MONTH,DATE)THEN 12

ELSE 0 END                                             AS FiscalPeriod,

                                YEAR(DATEADD(DAY,-1,DATEADD(MONTH, 1 + 12 -(DATEPART(MONTH,

DATE)- @FiscalStart + 1 +CASEWHEN @FiscalStart >

DATEPART(MONTH,DATE)THEN 12 ELSE 0 END),DATEADD(DAY,

(DATEPART(DAY,DATE)*-1)+1,Date))))

AS FiscalYearEnd,

                                CASEDATEPART(MONTH,DATE)- @FiscalStart + 1 +CASEWHEN

@FiscalStart >DATEPART(MONTH,DATE)THEN 12 ELSE 0 END

                                                                WHEN 1 THEN

WHEN 2 THEN

WHEN 3 THEN 1

                                                                WHEN 4 THEN

WHEN 5 THEN

WHEN 6 THEN 2

                                                                WHEN 7 THEN

WHEN 8 THEN

WHEN 9 THEN 3

                                                                WHEN 10 THEN

WHEN 11 THEN

WHEN 12 THEN 4

                                                                END                                         AS FiscalQuarter

 

                FROM

                CTE_DatesTable

 

                OPTION (MAXRECURSION 10000)

Again, you can simply copy and paste the script from THIS ATTACHMENT into query editor.  However, you need to set a parameter to tell the script when your fiscal year starts.  Towards the top of the script, you’ll see a command “SET @FiscalStart = “ and you need to enter a month number from 1 to 12 where 1 is January through to 12 being December.  Therefore, if the first month of your financial year is April, set this value to 4.  Otherwise, the script works the same as the previous example but returns some additional Fiscal Time elements.

And it’s as simple as that!  Thanks Scott – truly brilliant!