My Experiences with Hosted PowerPivot, Part One

February 21, 2012

Guest post by David Churchward

Pivotstream Dashboard Application

One of Azzurri Communications Ltd’s PowerPivot
Applications Running in the Browser

Six Months Ago:  The “Lightbulb” Moment

Almost exactly six months ago, after being a long time reader of this blog, I emailed Rob and asked him a question regarding something that I just couldn’t get my head around in DAX – Banding!  He kindly responded, and his answer solved my problem, so I asked him if I could return the favour somehow.  He asked if I’d mind writing it up as a guest post, which I did.

Now, double-digit guest posts later, I’m amazed at how far I’ve come in short order.  Something definitely “clicked” for me that day, and my grasp of PowerPivot’s capabilities expanded rapidly.  It felt like that moment that I imagine Pianists reach where they can suddenly play by ear, because whilst I could conquer most things in DAX, it didn’t seem to quite “flow” – until that day!

Up until that point I had viewed PowerPivot as a “private” tool – something that was useful for me in my work, a supplement to other tools and methods.  But starting six months ago, I started to understand that PowerPivot could, and SHOULD, be used to improve or replace most of our existing Business Intelligence and Analytics tools.

Step One:  Azzurri Deploys its First “On Premise” SharePoint Server

At Azzurri, I am fortunate to enjoy two critical flavours of support:

  1. Our executive team is open-minded to progressive ways of doing things (so long as there is a solid value proposition).
  2. My tech team is a crack outfit who will bring me the moon if I ask for it, but tend to make reasonable alternative suggestions such as building data warehouses.

In other words, Azzurri is the perfect sort of place to deploy PowerPivot for SharePoint, bordering on a textbook example.  There aren’t many companies of Azzurri’s size where I could explain the benefits of a PowerPivot server, win people over, and have a server deployed two weeks later.  But that is precisely what happened Smile

Step Two:  Start Emailing Rob Again Smile

We didn’t just deploy the server, we immediately began USING it for serious work.  And that led to questions – questions about performance.  Questions about hardware.  Questions about customisation.  Questions about refresh.  Questions about “core and thin” workbooks.

Rob and I had a friendly correspondence going at that point, so I started sending those questions over.  I even looped him into email chains with our tech team, and we talked through a number of issues and optimisations.

Step Three:  Try Out Hosted PowerPivot in Parallel

Everything I do in Systems Development, especially with my Finance background, is about Cost V Benefit, ROI, IRR and payback.  With this in mind, I started wondering whether it made sense for us to develop PowerPivot for SharePoint expertise of our own.

We had originally decided to go with our own SharePoint deployment because we had the required licences and a particularly clever team who I had every faith could deliver.  This seemed obvious as SharePoint was already in operation at Azzurri.  My initial view was that it must be relatively straight forward to bring all of the BI tools into the equation.

Two weeks into the process, however, I was already seeing that things might not be as straight forward as I had first hoped.  Performance was the first major barrier that I hit and I couldn’t be entirely sure what kind of investment in hardware might be required to alleviate this.  Out of nowhere, PowerPivot gallery started playing games which turned out to be an IE9 issue and then I was introduced to Kerberos which, it turns out, isn’t a breakfast cereal that I was yet to encounter!

I knew about the Pivotstream Hosted Solution of course, and I still wasn’t ready to commit to hosting, but I decided that running a trial in parallel made a lot of sense, especially since I was particularly aware that my tech team needed to be doing other things.

I’m very glad that we decided on a trial, because step four was to switch over full-time.

Goodbye “Do it Yourself”, Hello SaaS

The journey I’ve been on as a customer of Pivotstream has validated for me that the SaaS model together with the capability of PowerPivot makes for a more compelling business solution for reporting and BI than any alternative that I can find.

I’ve been particularly conscious of making sure that my tech team spend their time where they can really drive business value – building Data Warehouses, ETL and efficient business processes.  It was clear that time spent developing SharePoint Server was time not spent adding value elsewhere.  There’s no doubt in my mind that they would have delivered, but I knew that they could deliver more value elsewhere to more than offset any cost of hosting.

Summary of Our Experience

Once I had taken the decision to try out Pivotstream’s hosted solution, it became clear that “elapsed time” taken was no longer going to be a constraint to the project.  On that same day, Azzurri had it’s own Pivotstream site in full working order with admin and consumer accounts setup for the trial.  It was now down to me to start making this a fully functional dashboard.

Naturally, I had workbooks at the ready and I loaded a few up immediately.  I started sniffing some of the additional features that I could now start playing with.  Before I knew it, I was canvassing Pivotstream for direction on Query String URL filtering (an awesome attribute to drill across to other dashboards).  A handy guidance document found it’s way into my inbox and I was away.

I was supplied with a program to split core and thin workbooks, another gem that just saves time and aggravation.  I served up a query with web part layouts and, next day, I get a new page layout deployed straight to our site.

Immediately, the focus of what I was delivering was about end user usability as opposed to finding ways around potential (and in some cases very evident) performance issues.  Performance was immediately apparent on the Pivotstream solution, as could be immediately seen by some of my more “chunky” analysis that didn’t even make it flinch.  My in-house SharePoint Server could take upwards of a minute to open these workbooks whereas the hosted solution barely registers seconds.

Within a matter of days, I realised that the limits of this solution only existed to the extent of the limits of my imagination in creating dashboards.  All of a sudden, my focus was turned on making sure that full value was derived and, to that end, I started spreading the word within Azzurri.  Some initial training took place and I immediately recognised that these clever individuals that I was working with had even more insight bursting to get out and the fact that we were playing in Excel meant that they could immediately relate to what they were being shown.  I had hit that fantastic point in the project where momentum starts taking over and this is probably less than two weeks after starting the trial.

Speed (both of implementation and application), elimination of complexity and additional value adding applications delivered in a scalable data-centre model with an OPEX cost model sums it up for me.  Now, it’s just about making the dashboards deliver the real benefit to the business – insight!

As I’ve been writing this, Rob’s reminded me of a comment I made back towards when this whole thing started:

“My key driver is laziness so I’m always looking for quicker and better ways to do things. In doing so, I find myself working non-stop so I may have my driver wrong or I’m failing miserably!”

The reality is that Hosted PowerPivot does do it quicker and better.  I’m working non-stop because the results speak for themselves and I fundamentally “get it”.  The reality is that my driver was wrong!


Prepayments & Deferred Revenue

January 24, 2012

 

Guest Post by David Churchward 

Prepayments & Deferred Revenue Layout Example

You’ll often hear us Accountants referring to things like adjusting for timing differences or prepaying costs or deferring revenue.  This is often interpreted as “massaging the numbers”, but, believe me, there is a very reasonable theory behind it.  These sort of adjustments can be made for a number of reasons, but the main underlying concept is that adjustments have to be made to ensure that the transaction is reflected in the period to which it relates.

In this post, I’ll explain how prepayments and revenue deferrals work together with the DAX measures that drive these calculations.

What are Timing Differences?

It’s worthwhile just taking the time to understand what we’re trying to do here.  Take, for example, a support contract that you bill to your customer in advance for 12 months.  You invoice the customer in advance and the customer duly pays.  However, as per accounting principles, you have to reflect the fact that the invoice relates to 12 months worth of support.  In it’s simplest sense, your Profit and Loss should therefore see 1/12th of that revenue in each reporting month.  Since the invoice is for a full 12 months, we have to create transactions to defer the element that relates to future periods, entering the balance onto the Balance Sheet.

As a quick example of the calculation, at the end of the first month to which the support contract relates, you would defer 11/12ths of the invoice value:

Deferral Value = Invoice Value * (Number of Future Months / Total Months on Invoice)

We would therefore see 1/12th of the invoice value on our Profit & Loss and the remaining 11/12ths would be sat on our balance sheet.

Referring to Prepayments separately from Deferred Revenue (or Deferred Costs) for that matter is simply a matter of terminology.  The underlying concept of the calculation is the same.  I consider everything to be a Prepayment in this sense.  The only difference is where the transactions sit in the Profit & Loss and Balance Sheet statements.

Note – Accountants may defer revenue or costs for other reasons.  This post is simply a calculation to reflect the timing difference adjustment.

Days or Months

In the simplified example above, I’ve used a monthly concept that allows for a simple allocation of the transaction value to the Profit & Loss in equal monthly values.  However, as we know, the number of days in each month can differ.  For this calculation to be more accurate, we should pro-rata the calculation based on the number of days in that month.

The DAX measures that I show in this post will allow for selection of Monthly Deferral / Prepay or Daily Deferral / Prepay.  Which you use depends on your accounting policy.

Data Structure

My data structure is as follows:

Fact Table (Prepayments)

I have a core fact table which contains all transactions that require a prepayment or deferred revenue adjustment.  I have called this table Prepayments.

Prepayments Fact Table

The key elements to this table are as follows:

Date – This is the transaction date of the invoice posting

Nominal_PandL – This is the Profit & Loss Nominal Ledger account that the invoice was posted to

Nominal_Prepay_BS – This is the Balance Sheet Nominal Ledger account where the prepayment or deferral will be posted to.

Start / End – These are the start and end dates of the transaction.  If the invoice relates to a period from 1st Jan 2012 to 31st Dec 2012 then these dates would be 01/01/2012 and 31/12/2012 respectively.

Method – This is either MONTH or DAY depending on the level of granularity that you want your calculation to be calculated at.

PandL_Values – This is the total value of the invoice charge.

Prepay_Start / Prepay_End – I’ll come onto these alternative dates shortly.  In brief, these exist because your prepayment timeframe may differ from the strict dates to which the invoice relates.

Days Table

Prepayments Days Table

This table holds a sequential, unbroken list of dates (Date) together with associated attributes including Month End Date, Previous Month End Date, Month Start Date and Next Month Start Date.  These attributes become relevant in our measures and, whilst these can be calculated at run time, it’s easier to take care of them this way.  The purest would be correct in suggesting that there’s a potential performance impact, but dealing with these aspects in our measure may confuse the issue for now!

Months Table

Prepayments Months Table

Similar to the Days table, this table holds a collection of attributes relevant at the month level including Month End Date, Previous Month End Date, Month Start Date and Next Month Start Date.

Start Dates Table

Prepayments Start Dates Table

This is another sequential list of dates which is joined to the Start field on the Prepayments table.  I’ll come on to the PrepayStartdate field shortly.

End Dates Table

Prepayments End Dates Table

This is another sequential list of dates which is joined to the End field on the Prepayments table.  You’ll notice the absence of a third field which, again, I’ll come on to.

Table Relationships

Prepayments Table Relationships

The only relationships that exist are between the Prepayments table and the Start_Dates and End_Dates tables.

Dates That Span Periods

A charge period can be anything.  In it’s simplest form, it would be for complete months starting on the first day of a month and ending on the last day of a month.  However, things are never that simple.  Unfortunately, you could get a charge that is for 3 months from 14th Jan 2012 to 13th Apr 2012 as an example.  The time between these two dates is indeed 3 months, but the charge covers 4 discrete calendar months.

If you’re using a Days method of apportionment, this isn’t a problem.  However, if you’re apportioning on the basis of months, you have 3 options in this example:

  1. Take the 3 equal months charges as Feb, Mar and Apr
  2. Take the 3 equal months charges as Jan, Feb and Mar
  3. Take equal monthly charges for Feb and Mar with a pro-rata charge for Jan and Apr.

In truth, if you’re going to even attempt option 3, you would probably be better off jumping straight for the Days apportionment and circumvent the problem!  Options 1 and 2 depend on accounting policy and president.  The prudent option would probably be to go for option 2 for costs (taking them as early as possible) and option 1 for Revenue (taking later rather than earlier).  On the basis of consistency, I feel it’s best to prescribe this in your accounting policies and then use the same method for everything.  Having said that, the best option is almost certainly to take a Days approach.

For the purpose of this post, I’m going to use option 1.  This means that I’ll translate any start date that isn’t the first day of the month to the first day of next month.  I create two calculated fields on my Prepayments table as follows:

Prepay_Start

=IF(Prepayments[Method]=”MONTH”,RELATED(Start_Dates[PrepayStartdate]),Prepayments[Start])

 

Prepay_End

=IF(Prepayments[Method]=”MONTH”,RELATED(End_Dates[MonthEndDate]),Prepayments[End])

These measures retrieve an alternative date from the Start_Dates and End_Dates table.  For any mid-month dates, this table holds a PrepayStartdate of the first of the next month.

The Months Approach

Firstly, let’s just remember our simplified equation to give us the deferral value:

Deferral Value = Invoice Value * (Number of Future Months / Total Months on Invoice)

And now we’ll jump straight into the DAX

Month_Prepayment

=IF(COUNTROWS(VALUES(Months[MonthEndDate]))=1,

IF(LASTDATE(VALUES(Prepayments[Prepay_Start]))<LASTDATE(VALUES(Months[MonthEndDate]))

&&LASTDATE(VALUES(Prepayments[Prepay_End]))>LASTDATE(VALUES(Months[MonthEndDate])),

CALCULATE(

SUMX(Prepayments,

Prepayments[PandL_Value]

*(

(CALCULATE(COUNTROWS(Months),

DATESBETWEEN(

Months[MonthEndDate],

LASTDATE(VALUES(Months[MonthEndDate])),

LASTDATE(End_Dates[MonthEndDate])

         )

       )-1

     )/

CALCULATE(COUNTROWS(Months),

DATESBETWEEN(

Months[MonthEndDate],

LASTDATE(Start_Dates[PrepayStartdate]),

LASTDATE(End_Dates[MonthEndDate])

         )

   )

  )

),

Prepayments[Method]=”Month”

    )

)

)

The first two IF statements in this measure calculate when we want the rest of the measure to evaluate.  We want this measure to evaluate when:

  1. The number of month end dates on our report is 1 (because anything else is likely to return an error)
  2. The prepayment start and end dates fit within the range of our report.  This, essentially, tells the report which month end dates to display and the rest of the measure uses these dates to evaluate the correct result to display.

We then use a CALCULATE statement to calculate our basic equation:

Deferral Value = Invoice Value * (Number of Future Months / Total Months on Invoice)

Invoice Value = SUMX(Prepayments,Prepayments[PandL_Value]

Number of Future Months =

*(

(CALCULATE(COUNTROWS(Months),

DATESBETWEEN(

Months[MonthEndDate],

LASTDATE(VALUES(Months[MonthEndDate])),

LASTDATE(End_Dates[MonthEndDate])

         )

       )-1

    )

 

This calculates the number of month values in the Months table using the COUNTROWS function that sit between the MonthEndDate on our report and the MonthEndDate of the end date of our transaction.  I then subtract 1 to reflect that the MonthEndDate of the report is not included as it is the current reporting month.

Total Months on Invoice =

/

CALCULATE(COUNTROWS(Months),

DATESBETWEEN(

Months[MonthEndDate],

LASTDATE(Start_Dates[PrepayStartdate]),

LASTDATE(End_Dates[MonthEndDate])

         )

   )

Once again, I’m counting the number of entries in the Months table that sit between the start and end dates of our prepayment timeframe.

I then apply a filter to the CALCULATE statement to ensure that this measure is only applied to transactions where the Method is set to “MONTH”.

So why do we have to use SUMX for the Invoice Value?

The simple answer is that we need to conduct this equation at the transaction level of granularity and then SUM the results.  The main reason for this is that we’re using a division calculation which is only relevant at that level and becomes confused and meaningless when aggregated.

In order to show this, I’ll explain a simple example of converting USD to GBP with some theoretical exchange rates:

SUMX Example

What are the values for X and Y?  We obviously know that Y needs to be 150, being the sum of the two GBP values.  However, if we use SUM, we wouldn’t get that answer.  The answer that you would get depends on how you condition your measure but you wouldn’t get an answer of 150, it’s more likely to be 150/2=75 (where MAX is used in the equation) or 250/3.5=71.42 (where SUM is used) or 250/1.75=142.86 (where SUM and AVERAGE is used).  The correct value only comes about when you use SUMX which sums the underlying outcomes of the equation and evaluates at the granularity level of the table that you specify (in our case the base level fact table – Prepayments).

This is a very brief outline of a very complex and clever function.  In short, if you want to simply add the outcome of underlying equations, SUMX is the way to go.

It’s worth checking out one of my favourite ever posts to start getting to grips with SUMX – SUMX() – The 5 point palm exploding fxn technique.

The Days Approach

This uses the same methods and concepts as the Month approach.  The only difference is that we’re evaluating to the number of days in the relevant periods of time.

Days_Prepayment

=IF(COUNTROWS(VALUES(Months[MonthEndDate]))=1,

IF(LASTDATE(VALUES(Prepayments[Prepay_Start]))<LASTDATE(VALUES(Months[MonthEndDate]))

&&LASTDATE(VALUES(Prepayments[Prepay_End]))>LASTDATE(VALUES(Months[MonthEndDate])),

CALCULATE(

SUMX(Prepayments,

Prepayments[PandL_Value]

*(

(CALCULATE(COUNTROWS(Days),

DATESBETWEEN(

Days[Date],

LASTDATE(VALUES(Months[MonthEndDate])),

Prepayments[Prepay_End])

                    )-1

  )/

CALCULATE(COUNTROWS(Days),

DATESBETWEEN(

Days[Date],

Prepayments[Prepay_Start],

Prepayments[Prepay_End]

      )

             )

         )

      )

,Prepayments[Method]=”Day”

  )

       )

  )

Prepayments Total

We now have two measures to evaluate our Days Prepayments and our Months Prepayments.  We can bring these together in one measure as follows:

Prepayments_Total

=SUMX(Prepayments,[Month_Prepayment]+[Days_Prepayment])

Once again, I have to use SUMX to ensure that this measure sums the total of the underlying calculations.

Prepayments Without Values Not Prepaying

The “Values Not Prepaying”

So we’ve dealt with the reducing balances associated with Monthly and Daily Prepayments and Deferred Revenues.  However, what about a situation where the invoice charge relates to future months that haven’t yet been reached?  For example, we receive an invoice in January that relates to April and beyond, as in the screen shot above where the last entry has a transaction date of 10th Jan but relates to a period April to October.  In this instance, we have to prepay (or defer) the full amount of the invoice for January, February, March and April.

Value_Not_Prepaying

=CALCULATE

(

SUMX(Prepayments,

Prepayments[PandL_Value]

),

FILTER(Prepayments,

Prepayments[Date]<LASTDATE(VALUES(Months[MonthEndDate]))

            ),

FILTER(Prepayments,

Prepayments[Prepay_Start]>LASTDATE(VALUES(Months[MonthEndDate]))

  )

   )

In this measure, we once again use SUMX to ensure that we SUM the outcome of the underlying evaluations.  However, we apply two filters to limit the selection to situations where certain criteria hold.  This is an alternative to preceding the CALCULATE with an IF statement to conduct this evaluation first.

  1. The first filter evaluates when transaction date is less than the MonthEndDate of the report column
  2. The second filter evaluates when the Prepay_Start date is greater than the MonthEndDate of the report column.

We then add this into our Prepayments_Total measure as:

Prepayments_Total

=SUMX(Prepayments,[Month_Prepayment]+[Days_Prepayment]+[Value_Not_Prepaying])

 

Prepayments Balance Sheet Layout

And there you have it.  Our Balance Sheet layout above shows the total value of each transaction that needs to be prepaid or deferred in each month.

What This Means to Accountants

For those Accountants amongst you, you’ll no doubt have cottoned onto the fact that, if you’re careful, this could be your prepayments and deferred revenue reconciliations conquered.  The process that I’ve always implemented is as follows:

  1. Ensure that all transactions are posted to the Profit and Loss in full, as if the P&L was going to take the whole value in one period.
  2. Capture prepayment start and end dates on each transaction
  3. Extract the data into a PowerPivot application such as this.
  4. Refresh to give you the values that need to be prepaid (as per the above screenshot)
  5. Journal the appropriate values between the Profit and Loss and Balance Sheet on a reversing journal.
  6. Next month the previous transaction will reverse and you do the same again.

By doing this, you’ve got a self reconciling Balance Sheet for Prepayments and Revenue Deferrals (and all transactions for that matter that have a straight line release – for example, you could use this for elements of Fixed Asset Depreciation but that’s a whole other post).

And One More Thing

You may have noticed in the first graphic that there was also a Profit & Loss impact section to the report.  I’ll come onto this in a future post but it uses similar concepts to the measures in this post.  With this in place, you’ve got elements of a forecast P&L, you’re on course for Recurring Revenue Assurance analysis and you’re putting in place the building blocks of a full Finance Business Intelligence Application that’s just dying to be exposed in SharePoint as per Rob’s post In the Browser Aesthetics Yield a Greater Return.


Switching Subtotals On & Off Dynamically Using DAX in PowerPivot

January 5, 2012

Guest Post by David Churchward

Cash Flow Statement With and Without Subtotals

Subtotals on or off for a field - They’re the options that you get.  That’s probably great for a lot of situations, but I would put money on the majority of readers of this post having had a situation where they’ve asked “where’s the option to not display a subtotal when there’s only one record being shown?”.  Frustrating isn’t it?  You end up with a load of subtotals simply repeating the same value as it’s own one subset member because one of your other report elements has a subset of data that does need subtotalling.  You can, of course, toggle the show and hide for each member, but wouldn’t it be better if it was dynamic?

Fortunately, DAX has the answer.  It’s not the answer that I would have hoped for because this one feels like it should be an option setting somewhere.  However, we can deliver the same functionality by writing it into our measures.

In this post, I’ll explain how to create a dynamic measure that works out whether a subtotal is needed and displays that subtotal accordingly.  I’ll be using the report that was created as part of the Cash Flow Statement post.

The Subtotal Issue

Let’s refer back to the final report that I created for the Cash Flow Statement.

Cash Flow Statement With Totals2

In the column CF3_Name, the only subtotal that I want to see is the one on Working Capital as this is the only one that has any lower level detail.  You’ll notice that all other areas are carrying subtotals  which only serves to repeat the number above it.

COUNTROWS and COUNTX

In order to solve this problem, you obviously need to know how many subset elements there are that would make up your subtotal.  It would be reasonable to jump straight for COUNTROWS.  Let’s examine that quickly with a measure that we’ll call CF4_Headings_No in the hope that it will tell us how many headings are in our subset.

CF4_Headings_No

= COUNTROWS(VALUES(CF4[CF4]))

COUNTROWS Separate Tables

On the face of it, that doesn’t appear to give us what we want.  We have the same results for all headings and CF4_Name repeats all of the same headings.  The reason that this happens is because CF3_Name and CF4_Name are held on separate tables.  Although they hold relationships to the same table, PowerPivot doesn’t pre-determine which combinations of these fields exist, even at run-time.

So let’s look at what happens when these headings are held on the same table with the same COUNTROWS formula (but obviously changing table name etc!).

COUNTROWS Same Table

That looks better right?  We can see all of the correct combinations of CF3_Name and CF4_Name.  On the one hand, this is good because we can distinguish the fact that Working Capital is the only element that has more than 1 subset item and therefore the only element that requires a subtotal.

However, there’s one big downside here.  For all other sections, we can’t distinguish between a subtotal line and a detail line.  There’s a couple more downsides here too:

  1. I have to create a table that holds all of the combinations of these headings.  This isn’t the biggest issue as I have a table called GL_Headings that holds the codes associated to these headings.  I can create a RELATED()measure in my PowerPivot dataset to return the heading name.  However, why bother if we don’t have to?
  2. Don’t underestimate the power of the function COUNTROWS(VALUES(TABLE[FIELD])) returning a value of 1 which happens when headings are held as a separate table.  This opens up a whole host of functionality and it’s probably one of the most important aspects of PowerPivot in my opinion.

So, what’s the final analysis of COUNTROWS

  1. Using COUNTROWS against a separate table with unique values for each heading gives us the means to distinguish between subtotal sections and detail sections of our report.
  2. Using COUNTROWS against a combined table with heading combinations tells us the CF3 headings that carry detail where we do want to see a subtotal.

I’m sure you’ve probably deduced where I’m going with this as the combination of these two elements probably tells us what we need to know to figure out whether a subtotal is required.  Unfortunately, this is not entirely true.  The problem is that the combination of these elements won’t operate in context unless the context is directly applied against the heading table (ie you put a slicer on the report for the cash flow heading elements).  What about a date slicer?  If you apply a date slicer which is a separate table, these measures won’t react.

Bring on COUNTX

Let’s jump straight into the DAX on this one and see what it’s doing by creating a measure called CF3_Total_Reqd.

CF3_Total_Reqd

=COUNTX(CF4,[Cash_Flow_Statement])

COUNTX is analysing our CF4 table in the context of our measure [Cash_Flow_Statement] (as calculated in the Cash Flow Statement post).  The context applied to [Cash_Flow_Statement] is carried across to our COUNTX formula.  We run this against our CF4 table because we want to get the number of detail heading names that carry a value in our [Cash_Flow_Statement] measure.

So what do we get:

COUNTX

Our CF3_Total_Reqd measure using COUNTX is telling us how many headings are used in our dataset for the MonthEndDate shown on our report.  This is exactly what we want.  What’s even more special is that if, in a particular month, our [Cash_Flow_Statement] measure only has one heading in the Working Capital section, the Working Capital Total would return a value of 1, which is a scenario when we want to switch our subtotal off.

But there’s a problem

Our COUNTX measure, as it stands, is returning a count where we have a heading, but it isn’t telling us when we have a valid blank entry at the CF4_Name level.  As a result, any CF3 category that doesn’t carry CF4 heading detail, isn’t telling us anything.  I know we have Operating Profit elements, but we’re not getting any counts returned.

But what if we specify the column that we’re analysing in our CF4 table?  We can do this by adding in VALUES and specifying the column in our CF4 table.

CF3_Total_Reqd

=COUNTX(VALUES(CF4[CF4_Name]),[Cash_Flow_Statement])

COUNTX With VALUES()

We have a result!  And our Working Capital section is behaving itself too!

And now for the Subtotal Switch On / Off Measure

Consider our two measures:

  1. CF3_Total_Reqd this tells us when a total is required ie CF3_Total_Reqd > 1
  2. CF3_Total_Reqd also tells us where detailed elements exist, ie returns a value of 1, but this measure does the same for those associated subtotals.  CF4_Headings_No, at the same time, helps us distinguish, generally and irrespective of context, between detail and subtotal levels on our report ie CF3_Total_Reqd = 1 AND CF4_Headings_No = 1

If we combine these elements, we get a measure that we’ll call Cash_Flow_Heading_Tidy

Cash_Flow_Heading_Tidy

=IF(

[CF3_Total_Reqd]>1||

([CF3_Total_Reqd]=1&&[CF4_Headings_No]=1),

[Cash_Flow_Statement],

BLANK()

)

Cash Flow Statement Major Subtotals

That gives us what we want, right?  Well not quite.  The Cash Flow From Operations section is now right.  However, the other sections in my CF2_Name column have reached the criteria for their totals to be eliminated.  If I don’t want this effect (and I don’t because this is a major total), I have to specify for my Cash_Flow_Heading_Tidy to directly include those totals.

Cash_Flow_Heading_Tidy

=IF([CF3_Total_Reqd]>1||

([CF3_Total_Reqd]=1&&[CF4_Headings_No]=1)||

COUNTROWS(VALUES(CF3[CF3_Name]))>1,

[Cash_Flow_Statement],

BLANK()

)

Cash Flow Statement Complete

There you have it.  All subtotals behaving properly.


Cash Flow Statement in PowerPivot

December 15, 2011

 

Guest Post by David Churchward

Cash Flow Statement

There’s few, if any, reports that I consider to be more important than the Cash Flow Statement.  Every business needs one, probably above any other report.  There’s numerous different versions of Cash Flow reports and, in reality, it can become a very personal thing to a company so I’m not going to propose to provide a one size fits all report.  In addition, I’m not going to proclaim to provide an IAS7 – Cash Flow Statement (that’s International Accounting Standards) textbook report because this isn’t designed as an accountancy lesson (but, if you’re interested, the method I’ll show here relates to the Indirect Method in IAS7).

With that said, if you consider this post alongside Kasper’s excellent post Building a cash flow statement in PowerPivot using Dynamic Measures in DAX, you should have all the tools you need to produce any Cash Flow Statement that you require.

The Accountancy Bit (very quickly!)

Just before we start, a quick accountancy point to consider.  Kasper’s approach considers cash payment and receipt transactions.  My model uses all transactions in the general ledger with the exception of Bank transactions.  On the basis that double entry exists (ie every transaction has an equal and opposite effect), all transactions that aren’t the bank transactions must equate to the same value.  The reason we use the “other” side of the account is because this side of the transaction tells us what the cash transaction was for.  As a result, we should simply be able to aggregate transactions into Cash Flow categories.

This seems quite straight forward.  Unfortunately, there are a few exceptions that need to be handled.  I’m only going to explain one which should give you the means to adjust for others.

Depreciation is purely an accounting adjustment.  This has the effect of crediting the balance sheet and debiting the P&L.  The effect of crediting the balance sheet has the effect of reducing the value that will be shown as Fixed Asset additions.  We therefore net off the depreciation transaction as if it never happened and I’ll show this later in this post.

The Dataset

My dataset (FACT table) contains 3 columns:

  1. GL_Account– this is my general ledger code.  Every transaction that forms the trial balance will carry a general ledger code and this tells us what the transaction was for.
  2. Month_End_Date – this date tells us the financial period to which the transaction relates.  I would advise using the month end date of the financial period in which the transaction will be reported as I’m doing in this example. (or a period number as in Rob’s Greatest Formula In The World posts).  It should be noted that a transaction date may not be in the same calendar period as the reporting date so care should be taken.
  3. Amount– this is the transaction value.  Using normal accounting signage, this will be a positive value for debits and a negative value for credits.

My dataset contains all transactions processed on the system – this is PowerPivot, it can handle it – and this essentially forms the basis of the trial balance.  If you have a lot of old data and don’t want to include it all, you could substitute historic data with opening balance values for each general ledger code.  This simply aggregates a lot of old transactions into one balance (per general ledger code) for an opening balance date, whilst maintaining the overall integrity of a balanced trial balance and the associated values.

You may have a dataset that holds monthly values for each general ledger code as opposed to each individual transaction.  This will work equally well although you won’t have the means to dissect the Cash Flow Statement by subset records such as customer or project.

If your data is held as monthly columns, it’s worth running an UNPIVOT.  Whilst we could condition PowerPivot to work with individual values for each period, it’s more efficient to work with one value column with associated period dates held in another column.

Related Tables

I link my my FACT table to a GL_ACCOUNTS table on the GL_Account code.  The GL_Accounts table provides details relating to the GL_Code such as account name and  report heading codes.  These heading codes, in turn, link through to a further GL_Headings table which holds cash flow report heading codes.  This, in turn, links through to cash flow heading tables.  It’s these headings that we will use on our Cash Flow Statement report.

Cash Flow Statement Dataset Schema

image

You’ll notice that CF1 doesn’t link to any other tables.  I’ll come onto that shortly.

General Ledger Account Mappings

The GL_Headings mapping is the critical element to this dataset.  This provides all of the key relationships between a general ledger code and the Cash Flow Statement report heading that you want the account to be represented under.

GL_Headings Table

image

My GL_Accounts map through to my Heading_Code on this GL_Headings table.  This table holds all of my key report headings and provides a useful grouping of accounts.  For instance, all general ledger codes that relate to Sales will carry a Heading_Code of 1010.

These definitions are important.  This is a one-off setup and you’ve probably done this indirectly in some way or another to drive any reports or Excel analysis that you’re currently doing (probably with VLOOKUP).  This process simply affirms that mapping.  Ideally, you would code these heading codes into your General Ledger account code setup and formulate the code allocation as part of the process to set up new general ledger accounts.

You’ll recall that I mentioned Depreciation earlier in this post.  Whilst this is a P&L account (heading code 1045 in this example), I’m coding this to exactly the same place as Fixed Assets on the Balance Sheet (heading code 2010).  This effectively eliminates the deprecation transactions.  An alternative would be to filter out the accounts in question.  You can do this, but be careful to ensure that those accounts net to zero.

Each of these codes map through to Cash Flow Statement heading tables.  These heading tables can be whatever you need them to be, with headings and ordering assigned however you want your final report to be represented.  The one’s I’m using are shown below.

CF2 – Cash Flow Statement Heading 2

image

This is my first grouping of headings.  These align with the IAS7 headings for CF2 codes 1, 2 and 3 and basically split my Cash Flow Statement between Operations Activities, Investing Activities and Financing Activities.

There’s a fourth section called Bank.  The Bank general ledger codes map through to this code and we eliminate them for the Cash Flow element of this report.  However, we bring them back into action when we display the Bank Balances at the end of the report.

CF3 – Cash Flow Statement Heading 3

image

This is the next level of granularity on my report.  The Bank general ledger codes don’t have any CF3 mapping and yet I’m carrying Opening and Closing Bank Balance codes on this heading set.  I’ll come onto why that is a bit later.  You’ll also notice that I’ve coded these headings with an Open_Bank  and Close_Bank flag.  We’ll use dynamic DAX measures later on to utilise these.

Note – there should NOT be any mapping of accounts to the CF3 codes 10 and 11 in this table (ie Open and Close Bank Balance).  We’ll use these simply to drive a DAX measure later.

CF4 – Cash Flow Statement Heading 4

image

One element of my Cash Flow Statement will carry a further breakdown – Working Capital (although you can have more if you want).  I’ve done this simply to provide an extra level of granularity, but you could decide to replace the Working Capital heading in CF3 with these headings and map accordingly.

I’ve got a few reasons for doing this:

  1. I like to show my Working Capital elements specifically – this targets the key operational balance sheet items that generally need to be tracked
  2. This helps to line up a future post!

CF1 – Cash Flow Statement Heading 1

You might, quite rightly, question why my ordering has gone to pot and I’m explaining CF1 last!  Quite simply, this table sits out on it’s own because it simply drives report sections as opposed to being a mapped heading code.  I’m going to show a Cash Flow section and then, below that, I’m going to show the effect on the bank balance.  No relationships required.

image

And Now for the DAX – Cash Flow

I’ve got to be honest, this measure isn’t going to set the world alight.  However, this goes to show how easy PowerPivot can be, especially if you condition your data properly.  I’m creating a measure aptly called Cash_Flow

Cash_Flow

=CALCULATE(

SUM(FACT[Amount]),

CF2[CF2_Name]<>”BANK”

       )*-1

This is a simple CALCULATE function which sums the [Amount] field on the FACT table with a simple filter set to ignore any GL_Accounts that have a CF2 category mapping of “BANK”.

You’ll notice that I multiply this by –1 to reverse the signage.  In simple terms this is because of the double entry effect.  We’re using the “other” side of the transaction which means that the sign convention is going to be back to front.  This simply reverses that effect.

When I pull this measure onto my report together with the CF1,2,3,4 heading names, I get the following.

Cash Flow in Excel - PowerPivot Turns Complex into Simple

You might say that this is job done (especially if you remove CF1 heading from the report). This does give us a Cash Flow Statement. However, this doesn’t give my bank balances and, if I include my CF1 heading, my report is repeating the same data for both of the CF1_Name headings. This is understandable as CF1 is not linked to my FACT table in anyway.  This is intentional (believe it or not!).

Bank Balances

I want to show opening and closing bank balances for the Bank section of my report and this is where it gets a bit more interesting.  Let’s create a measure that calculates these balances and let’s call this measure Bank…

Bank

=IF(COUNTROWS(VALUES(CF3[CF3]))=1,

IF(VALUES(CF3[Close_Bank])=1,

CALCULATE(

SUM(FACT[Amount]),

ALL(CF3),

CF2[CF2_Name]=”BANK”,

FILTER(

ALL(Month_End_Dates[MonthEndDate]),

Month_End_Dates[MonthEndDate]<=

MAX(Month_End_Dates[MonthEndDate])

            )

          ),

IF(VALUES(CF3[Open_Bank])=1,

CALCULATE(

SUM(FACT[Amount]),

ALL(CF3),

CF2[CF2_Name]=”BANK”,

FILTER(

ALL(Month_End_Dates[MonthEndDate]),

Month_End_Dates[MonthEndDate]<=

EOMONTH(MAX(Month_End_Dates[MonthEndDate]),-1)

  )

),

BLANK()

   )

    )

      )

We should be offering prizes for those of you would can instantly spot the GFITW elements!!  Anyone would think that this had somehow been contrived!  Believe me – it hasn’t.  This is a concept that I use regularly and it’s a joy to see Rob explain it so well so I don’t have to!

So what is this doing?  I’ll explain as clearly as I can, but it’s worth paying very close attention to Rob’s Greatest Formula In The World posts.  I won’t re-iterate too much here.

  1. The first IF statement simply serves to provide an answer as to where I want the measure to be evaluated.  I want an answer when my CF3_Heading is flagged as Open_Bank or Close_Bank.
  2. The second IF statement creates a value when my CF3_Heading [Close_Bank] is flagged.  Where this is the case, the following happens:
    1. Sum the [Amount] Field
    2. Expand my dataset to all CF3 headings in order to release the dependency on the current CF3 heading being evaluated – ALL(CF3)
    3. Apply a filter to include all of my Bank general ledger codes – CF2[CF3_Name]=”BANK”
    4. Apply a further filter (FILTER) that:
      1. Removes the dependency on the MonthEndDate – ALL(Month_End_Dates[MonthEndDate])
      2. Apply a filter that includes all months that pre-date and include the reporting month – Month_End_Dates[MonthEndDate]<=MAX(Month_End_Dates[MonthEndDate]) – this is important because the transaction base for the month won’t give us the balance.  We need to include ALL transactions that pre-date and include the reporting date.
  3. The third IF statement has a similar effect to the second IF statement, but it produces an opening balance and therefore the current reporting month has to be ignored.  As a result, I want to sum all transactions where the date is less than or equal to my prior month date – Month_End_Dates[MonthEndDate]<=EOMONTH(MAX(Month_End_Dates[MonthEndDate]) ,-1)

Cash Flow in Excel with Bank Balances Thanks to PowerPivot

Once again, we have the correct results but we’re getting values repeated for both CF1 headings.

Sorting Those CF1 Headings

We now have our two base measures.  All we now need to do is evaluate when to use each one.  Let’s create another measure called Cash_Flow_Statement.

Cash_Flow_Statement

=IF(COUNTROWS(VALUES(CF1[CF1]))=1,

IF(VALUES(CF1[CF1])=1,

[Cash_Flow],

[Bank]

   )

)

This measure simply evaluates for CF1 headings and determines that if my CF1 Heading Code is 1 (ie Cash Flow) then use my [Cash_Flow] measure.  Otherwise, use my [BANK] measure.

Cash Flow Finished

A bit of formatting and there you have it.  You’re just about done.  Or rather, you’re done if you think that’s enough.  I, however, don’t like some of these totals and duplication of values in certain areas.  There’s a way to deal with those totals but that’s a topic for a future post.

So what else can we do with this?

Just as a taster, here’s something that I like to call a Cash Flow Bridge.  This could be a topic for a future post, but just to whet your appetite……

Cash Flow Statement Bridge

If there’s enough call for this, I’ll get this scheduled in for a future post.  Neat eh!


Using Query Editor to Create a Time Table

December 1, 2011

 

Guest Post by David Churchward with Technical Input from Scott Senkeresty

The first thing I ever do when I create a new PowerPivot workbook is create the date tables that I need.  Until now, I’ve used a preconditioned SQL table to draw this information from.  Now, thanks to Scott Senkeresty, you can now create your own – from PowerPivot!

Scott is a true genius.  He spends a lot of time under the hood of Pivotstream operations working his magic and wizardry and, every now and then, he pops up for air to deliver a masterpiece like this.

 

It’s as Simple as Copy and Paste

If you read my last post SQL UNPIVOT Makes My Data Skinny, you’ll know where query editor is and the sort of tricks that it opens up within PowerPivot.  In this post, there’s some SQL scripts that create time tables and you can simply cut and paste them into the query editor and use them straight away.

Having said this, it’s worth checking out Rob’s post The Ultimate Date Table to see why a date table is important and where you can get one of your own from Azure DataMarket.  However, I think we’ve gone one step better now and given you a script that you can parameterise easily, condition relatively easily and setup extremely easily.  Add to that the fact that it runs in mere seconds!

AND there’s one more bonus, especially for Accountants, you can set this up for YOUR Fiscal Year!

Copy, Paste, Save – Simples!

To get going, simply copy and paste the script from THIS ATTACHMENT into query editor.  When you save it – bam – time table complete.

DECLARE @FirstDate DATE

DECLARE @LastDate DATE

SET @FirstDate =’1 Jan 2008′– Enter first date of calendar as ‘dd mmm yyyy’

SET @LastDate =’31 Dec 2015′– Enter last date of calendar as ‘dd mmm yyyy’

 

IF @FirstDate ISNULLOR @FirstDate =

                BEGIN

                                SET @FirstDate =’1 Jan 2000′

                END

               

IF @LastDate ISNULLOR @LastDate =

                OR @LastDate < @FirstDate

                ORDATEDIFF(DAY, @FirstDate, @LastDate)> 10000

                BEGIN

                                SET @LastDate =DATEADD(DAY, 10000, @FirstDate)

                END

 

                ;WITH CTE_DatesTable

                AS

                (

                  SELECT @FirstDate AS [Date]

         UNIONALL

         SELECTDATEADD(DAY, 1, [Date])

         FROM CTE_DatesTable

         WHEREDATEADD(DAY, 1, [Date])<= @LastDate

                )

 

                SELECT

                                Date,

                                DATEPART(DAYOFYEAR,Date)                             ASDayOfYear,

                                DATEPART(DAY,Date)                                          ASDay,

                                DATEPART(MONTH,Date)                                    ASMonth,

                                DATEPART(YEAR,Date)                                        ASYear,

                                DATENAME(WEEKDAY,Date)                               ASDayOfWeek,

                                LEFT(DATENAME(WEEKDAY,Date),3)                 AS DayOfWeekAbr,

                                DATEPART(WEEK,Date)                                       AS WeekOfYear,

                                DATEPART(QUARTER,Date)                                 ASQuarter,

                                DATENAME(MONTH,Date)                                  ASMonthName,

                                LEFT(DATENAME(MM,Date),3)                           AS MonthNameAbr,

                                DATEADD(DAY,-1,DATEADD(MONTH, 1,DATEADD(DAY,(DATEPART(DAY,

DATE)*-1)+1,Date)))                         AS MonthEndDate

 

                FROM

                CTE_DatesTable

 

                OPTION (MAXRECURSION 10000)

Without doing anything to this script, you’ll get a time table that spans Jan 2008 to Dec 2015.  That could be enough.  In which case, you’re done.  However, I’m sure you want more.

Change Date Parameters to Widen the Timeframe

The great news with this script is that it’s so easy to adjust to cover the timeframe that you need.  You just need to follow these simple couple of steps:

  1. Towards the top of the script you’ll see “SET @FirstDate = “.  You simply need to change the value to the first date in your dataset (or perhaps an earlier date to be sure).
  2. I’m sure you’ve already guessed step 2 – change “SET @LastDate = “ to a value that is beyond the last date in your dataset.

I almost wish I could make it more difficult but Scott’s nailed this so well that it’s almost impossible to get wrong – well almost!

A Quick Health Warning

This script uses a recursive CTE (Common Table Expression).  If you’re not careful, it’s very possible to spin this baby into an infinite cycle.  Without going into why this happens, here’s a couple of points to make sure that you don’t fall into that trap:

  1. The IF – BEGIN – SET – END statements are there to ensure that valid entries are made for @FirstDate and @LastDate.  It’s probably best to leave these in.
  2. At the end of the script, there’s a command OPTION(MAXRECURSION 10000).  This ensures that the script won’t loop more than 10,000 times (ie 10,000 days).  If you set the recursion to zero, this removes all limits and then you could get into an infinite cycle.

Don’t be scared to give it a go.  There’s controls here to stop anything going wrong and what’s the worst that could happen anyway!

The Next Level – Fiscal Elements

In all my time using SQL and PowerPivot, I’m yet to work in an environment where a year runs from 1st January to 31st December.  As a result, we refer to a Fiscal Year to represent this alternate calendar.  I took the liberty of hacking Scott’s script to add these elements in.

DECLARE @FirstDate DATE

DECLARE @LastDate DATE

DECLARE @FiscalStart INT

SET @FirstDate =’1 Jan 2008′– Enter first date of calendar as ‘dd mmm yyyy’

SET @LastDate =’31 Dec 2015′– Enter last date of calendar as ‘dd mmm yyyy’

SET @FiscalStart = 4 – First calendar period of Fical Year – 1=Jan, 2=Feb…., 12=Dec

 

IF @FirstDate ISNULLOR @FirstDate =

                BEGIN

                                SET @FirstDate =’1 Jan 2000′

                END

               

IF @LastDate ISNULLOR @LastDate =

                OR @LastDate < @FirstDate

                ORDATEDIFF(DAY, @FirstDate, @LastDate)> 10000

                BEGIN

                                SET @LastDate =DATEADD(DAY, 10000, @FirstDate)

                END

 

                ;WITH CTE_DatesTable

                AS

                (

                  SELECT @FirstDate AS [Date]

                 UNIONALL

                 SELECTDATEADD(DAY, 1, [Date])

                 FROM CTE_DatesTable

                 WHEREDATEADD(DAY, 1, [Date])<= @LastDate

                )

 

                SELECT

                                Date,

                                DATEPART(DAYOFYEAR,Date)                             ASDayOfYear,

                                DATEPART(DAY,Date)                                          ASDay,

                                DATEPART(MONTH,Date)                                    ASMonth,

                                DATEPART(YEAR,Date)                                        ASYear,

                                DATENAME(WEEKDAY,Date)                               ASDayOfWeek,

                                LEFT(DATENAME(WEEKDAY,Date),3)                 AS DayOfWeekAbr,

                                DATEPART(WEEK,Date)                                       AS WeekOfYear,

                                DATEPART(QUARTER,Date)                                 ASQuarter,

                                DATENAME(MONTH,Date)                                  ASMonthName,

                                LEFT(DATENAME(MM,Date),3)                           AS MonthNameAbr,

                                DATEADD(DAY,-1,DATEADD(MONTH, 1,DATEADD(DAY,(DATEPART(DAY,

DATE)*-1)+1,Date)))                         AS MonthEndDate,

                                DATEADD(DAY,-1,DATEADD(MONTH, 1 + 12 -(DATEPART(MONTH,DATE)

- @FiscalStart + 1 +CASEWHEN @FiscalStart >DATEPART(MONTH,DATE)

THEN 12 ELSE 0 END),DATEADD(DAY,(DATEPART(DAY,DATE)*-1)+1,

Date)))                                                   AS YearEndDate,

                                DATEPART(MONTH,DATE)- @FiscalStart + 1 +

                                                CASEWHEN @FiscalStart >DATEPART(MONTH,DATE)THEN 12

ELSE 0 END                                             AS FiscalPeriod,

                                YEAR(DATEADD(DAY,-1,DATEADD(MONTH, 1 + 12 -(DATEPART(MONTH,

DATE)- @FiscalStart + 1 +CASEWHEN @FiscalStart >

DATEPART(MONTH,DATE)THEN 12 ELSE 0 END),DATEADD(DAY,

(DATEPART(DAY,DATE)*-1)+1,Date))))

AS FiscalYearEnd,

                                CASEDATEPART(MONTH,DATE)- @FiscalStart + 1 +CASEWHEN

@FiscalStart >DATEPART(MONTH,DATE)THEN 12 ELSE 0 END

                                                                WHEN 1 THEN

WHEN 2 THEN

WHEN 3 THEN 1

                                                                WHEN 4 THEN

WHEN 5 THEN

WHEN 6 THEN 2

                                                                WHEN 7 THEN

WHEN 8 THEN

WHEN 9 THEN 3

                                                                WHEN 10 THEN

WHEN 11 THEN

WHEN 12 THEN 4

                                                                END                                         AS FiscalQuarter

 

                FROM

                CTE_DatesTable

 

                OPTION (MAXRECURSION 10000)

Again, you can simply copy and paste the script from THIS ATTACHMENT into query editor.  However, you need to set a parameter to tell the script when your fiscal year starts.  Towards the top of the script, you’ll see a command “SET @FiscalStart = “ and you need to enter a month number from 1 to 12 where 1 is January through to 12 being December.  Therefore, if the first month of your financial year is April, set this value to 4.  Otherwise, the script works the same as the previous example but returns some additional Fiscal Time elements.

And it’s as simple as that!  Thanks Scott – truly brilliant!


SQL UNPIVOT Makes My Data Skinny

November 29, 2011

 

Guest post by David Churchward

If you’re anything like me, you will have eagerly sucked data into PowerPivot, keen to get cracking with solving all of your reporting needs.  If you’re an Accountant (amongst others) doing this, you will no doubt have brought your trial balance dataset in and found endless columns of values representing monthly values.  “How do I get time intelligence to work with this?” is probably one of the first issues you’ve encountered and then, further down the line, when you’ve conjured up complex measures to resolve this problem, the words “why can’t I get this to run quicker” pass your lips.  On this note, everyone should pay close attention to Less Columns, More Rows = More Speed! where Rob proves that the tall and skinny table is much more effective than a short and wide version.

In this post, I’ll explain how SQL can solve this problem by transforming the data on the data extraction process using a SQL function call UNPIVOT and it’s available to everyone – not just developers!

This is PowerPivot using Pivot tables – why on earth do I want to UNPIVOT.  Just like a kid who ties their shoelaces wrong, Mum has to untie in order to do it properly!  The fact is that data transformation has to happen in many situations to leverage the full power that PowerPivot can provide.

What’s this SQL Stuff – Surely that’s for Developers?

I was reading Rob’s recent posts on Why PowerPivot is Better Fed From a Database and it made me realise that there’s a grey area that represents how far an Excel Pro can go in data transformation.  I’m going to leave the “Developer V Excel Pro” debate to Rob (but believe me, he’s right).  As an Accountant who learnt SQL but who is an Excel Pro through and through, my view is that Excel Pros should be able to transform their data to the extent that a SQL view can.

Some of you will be aware, but others won’t, that there is a SQL query editor in PowerPivot.  If I’m upsetting Developers by revealing this, I’m sorry, but, if you’re worried, please take this as an opportunity to provide your Excel Pros with a secure Dev environment!

Accessing The SQL Query Editor

You can access query editor either when you first import data or by editing a table that you’ve already imported.

Accessing Query Editor on First Import

  1. From the PowerPivot window, select From Database > From SQL Serverand set up your connection as you would do normally.
  2. Click Next and select the option Write a query that will specify the data to import

image

Accessing Query Editor From a Table Already Imported

  1. Select the table that you want to affect in the PowerPivot Window and select Design > Table Properties
  2. Change from Table Preview to Query Editor

image

image

And there you have it.  A script that is running to import the data that you’ve specified.  We can now edit this to transform the data.

Before I go ahead, I just want to set a benchmark regarding the data that I’m using.  When I’ve brought this data in, I’ve got 38,183 records using 14 columns.  We’ll see what the UNPIVOT does to this in the end.  Everyone’s expecting a lot more rows but fewer columns, right?

image

The SQL “UNPIVOT” SCRIPT

So, here it is, the script to use in Query Editor mode:

SELECT

– These are the fields that I want in my final dataset

NCODE,

NNAME,

VALUE_CATEGORY, – NEW FIELD 1 – The field that takes my value column names

VALUE – NEW FIELD 2 – The field that takes my values from those columns

FROM

– I specify where these fields come from

(

SELECT

NCODE,

NNAME,

NTURNOVER_C1, – Value Field

NTURNOVER_C2, – Value Field

NTURNOVER_C3, — Value Field

NTURNOVER_C4, – Value Field

NTURNOVER_C5, – Value Field

NTURNOVER_C6, – Value Field

NTURNOVER_C7, – Value Field

NTURNOVER_C8, – Value Field

NTURNOVER_C9, – Value Field

NTURNOVER_C10, – Value Field

NTURNOVER_C11, — Value Field

NTURNOVER_C12 – Value Field

 

FROM dbo.AZZ_V_POWERPIVOTPRO_TB –Substitute this with the name of your source table or view

) PIVOT_TABLE

– Unpivot the table               

UNPIVOT

(VALUE FOR VALUE_CATEGORY in        – These are the names given to the

       — value field (NEW FIELD 1) and value column name (NEW FIELD 2)

(

– Enter value fields as per the select above

NTURNOVER_C1,

NTURNOVER_C2,

NTURNOVER_C3,

NTURNOVER_C4,

NTURNOVER_C5,

NTURNOVER_C6,

NTURNOVER_C7,

NTURNOVER_C8,

NTURNOVER_C9,

NTURNOVER_C10,

NTURNOVER_C11,

NTURNOVER_C12

)

) AS UNPVT

WHERE VALUE <> 0

ORDER BY NCODE, VALUE_CATEGORY

Note – green text within the script are comments to provide some direction as you use this script.  You can copy these into query editor too – PowerPivot will ignore them.

How to Use This Script

  1. Copy and paste the script into your query editor
  2. In the first SELECT, enter the field names that you require in your final dataset.  These should correlate to fields in the table except where I have noted “NEW FIELD 1” and “NEW FIELD 2”.  These “NEW FIELDS” will carry the value elements that we are unpivoting.
  3. In the second SELECT, detail the fields that you have already detailed in the first select together with the column names carrying the values that you will unpivot.  Also change the table or view name in the FROM clause to reference the source data that you are using.
  4. In the UNPIVOT section, detail all of the value fields again that you are unpivoting.
  5. In the ORDER BY clause, select the fields that you wish to use for dataset ordering.

When you save this, your dataset will have been narrowed and you’ll have a new field denoting the data type (or value column reference).

image

You’ll notice that we’re down from 38,183 records to 7,103.  This won’t always happen.  The reality is that we’re able to strip out the zero value cells by using the WHERE VALUE <> 0 clause.  If you’re carrying a number of cells that are zero valued, we won’t see that item as a record in this new dataset.  As a result, we get fewer records in this case.  In any event, your data is now a lot more streamlined and ready for more efficient measures to be written.  You don’t have to include the WHERE clause if you prefer, for whatever reason, to bring all records in.  If that’s the case, just delete that clause.

One Further Step

To make this truly useful, we need to translate VALUE_CATEGORY into something more meaningful.  I would normally expect this to be a date but most trial balances use column names such as “CURRENT1” so that the system isn’t date dependent.  Another table would be required that the VALUE_CATEGORY column can link through to which translates that value, thereby opening up the whole time intelligence functionality.

A Quick Warning

This UNPIVOT approach has given you the power to transform your data into a more streamlined dataset.  However, by using this approach you’ve created a local version of the script.  You won’t be able to reuse this without copying and pasting it into other workbooks (or at least I haven’t found a way).  However, those Developer friends of ours can give you a centralised version that can be controlled and reused more easily.  If you ask these guys nicely, they might create a view or stored proc to do this for you.  I’ve found with my team that salt based snacks, chocolate and the occasional Spag Bol for lunch does the trick and, if it doesn’t, I go ahead and do it anyway!


Trended Moving Averages

November 8, 2011

Guest post by David Churchward

image

I’ve always been a firm believer that moving averages probably give a better insight into trends within a business than a simple trend line associated to a set of values such as monthly sales (although I tend to review these two values together).  The reason for this is that a trend can be skewed by one or two values that may not be representative of the underlying business such as spikes associated to seasonality or a specific event.  When BillD highlighted a query regarding this concept in his comments on Profit & Loss (Part 2) – Compare and Analyse, I thought it would be a great idea to flex our P&L dataset to provide some Moving Average capability.

In this post, I will explain what moving averages are intended to deliver and explain how to calculate them using the sales elements of the example data used in the Profit & Loss series of posts.  I will then add the flexibility for users to select the time frame that the moving average calculation should consider, the number of trend periods to be displayed and the end date of the report.

What is a Moving Average?

The most common moving average measure is generally referred to as a 12 month moving average.  In the case of our sales data, for any given period, this measure would sum the last 12 months of sales preceding and including the month being analysed and then divide by 12 to show an average sales value for that timeframe.  In financial terms, the equation is therefore quite simply:

12 Month Moving Average = Sum of Sales for Last 12 Months / 12

This all seems very straight forward but there’s a lot of complexity involved if we want to put the Moving Average timeframe (represented as 12 in the above example) in the hands of the user, give them the power to select the number of trend periods to be displayed and the month that the report should display up to.

The Dataset

The dataset that we’re using looks something like below.

image

Note – I’m using PowerPivot V1.  Design viewer is available in V2 but I’ve hashed this together – nothing clever!

You’ll notice that FACT_Tran (our dataset to be analysed) is linked to DIM_Heading1, DIM_Heading2 and DIM_DataType to provide some categorisation to our dataset.  I’ve also linked to Dates which is a sequential set of dates that more than covers the timespan of our dataset.  This table carries some static additional information based on the date:

Date_Month_End = EOMONTH(Dates[Date],0)

Date_Next_Month_Start = Dates[Date_Month_End]+1

Once again, we’re not quite registering on Rob’s spicy scale!  Rest assured that you’ll be getting a more intense DAX workout as we go on.

As these date measures aren’t expected to be dynamic, I’ve coded them in the PowerPivot window.  This allows them to be calculated on file refresh but they won’t need to recalculate for each slicer operation which removes performance overhead from our ultimate dynamic measure.

For reasons that I’ll come on to later, I also need the month end date on my fact table as I can’t use the Month End Date on my Dates table in my measures.  I can however pull the same value across to my FACT_Tran table using the following measure:

Fact_Month_End_Date = RELATED(Dates[Date_Month_End])

So What Are These Unlinked MA_ Tables?

The reason for these tables should become apparent as we go on.  In brief, they’re going to be used as parameters or headings on our report.  The reason that they exist and that they’re not linked to the rest of our data is simply because I don’t want them to be filtered by our measures.  Instead, I want them to drive the filtering.

Initial PivotTable Setup

I’m going to be displaying a series of data organised in monthly columns.  The user will be given slicers to set Month End Date (the last period to be shown on the report), Number of Periods for Moving Average (this will ultimately be part of our divisor calculation) and Number of Periods for Trend (this will be the number of monthly columns that we will display on our trend).  We can establish these slicers straight away and link them to the pivot.

I obviously need a month end date as a column heading but which one?  To some extent I’ve given this away earlier on.  In short, I need to use my MA_Dates[Month_End_Date] field.  The reason is that this field isn’t linked to our dataset and therefore won’t be affected by any other filters.  If I use a date field that is part of my dataset or part of a linked table, the values available may be filtered down by the users selections.  I can get around this using an ALL() expression to give me the correct values, but the problem is that the column is still filtered and my results will all be displayed in one column.  It’s difficult to explain until you see it so please go ahead and try – it’s worth hitting the brick wall to really understand it!

Calculating Sum of Sales for Last X Months

The first part of our equation is to calculate the total value for sales across all periods within a dynamic timeframe to be selected by the user.  For this I use a Calculate function that looks like this:

CALCULATE(

[Cascade_Value_All],

DIM_Heading1[Heading1_Name]=”Sales”,

DIM_DataType[Data_Type_Name]=”Actual”,

DATESBETWEEN(

Dates[Date],

DATEADD(

LASTDATE(VALUES(MA_Dates[Next_Month_Start_Date])),

MAX(MA_Function_Periods[Moving_Average_No_Periods])*-1,MONTH

   ),

LASTDATE(VALUES(MA_Dates[Month_End_Date]))

         )

  )

I’m using a base measure called Cascade_Value_All that was created in Profit & Loss – The Art of the Cascading Subtotal.  I’m then filtering that measure to limit my dataset to records that relate to Sales and a data type of Actual (ie eliminating Budget).  This is simple filtering of a CALCULATE function.  However, it gets a bit more tasty with the third filter which limits the dataset to a series of dates that are dependent on the users selections in slicers and our date column heading.

The DATESBETWEEN function has the syntax DATESBETWEEN(dates, start_date, end_date) and works like this:

  1. I set the field that requires filtering (Dates[Data]).  I’ve found that this works best if this is a linked table of sequential dates without any breaks.  If you have any breaks, there’s a chance you might not get an answer as the answer that you evaluate to has to be available in the table.
  2. My start date is a DATEADD function that calculates the column heading date less the number of months that the user has selected on the “Moving Average No of Periods” slicer.  I use the LASTDATE(VALUES(MA_Dates[Next_Month_Start_Date)) function to retrieve the Next_Month_Start_Date value from the MA_Dates table that relates to the date represented on the column heading.  I then rewind by the number of months selected on the slicer using MAX(MA_Function_Periods[Moving_Average_No_Periods])*-1.  The “-1” is used to go back in time.  The reason I use Next_Month_Start_Date and a multiple of –1 is more clearly explained in Slicers For Selecting Last “X” Periods.
  3. My end date is simply the Month_End_Date as shown on the column heading of the report.  This is calculated using LASTDATE(VALUES(MA_Dates[Month_End_Date]).

image

That’s great, but my measure isn’t taking any account of my “Show Periods Up To” selection and the “Trend No of Periods” that I’ve selected.  We therefore need to limit the measure to only execute when certain parameters hold as true based on these selections.  I only want values to be displayed when my column heading date is:

  1. Less than or equal to the selected Month End Date on my “Show Periods Up To” slicer AND
  2. Greater than or equal to the selected Month End Date LESS the selected number of periods on my “Trend No of Periods” slicer.

To do this, I use an IF statement to determine when my CALCULATE function should execute.  Let’s call this measure Sales_Moving_Average_Total_Value

Sales_Moving_Average_Total_Value

= IF(COUNTROWS(VALUES(MA_Dates[Month_End_Date]))=1,

    IF(VALUES(MA_Dates[Month_End_Date])<=

LASTDATE(Dates[Date_Month_End])

&&VALUES(MA_Dates[Month_End_Date])>=

DATEADD(

LASTDATE(Dates[Date_Next_Month_Start]),

(MAX(MA_Trend_Periods[Trend_Periods])*-1),MONTH),

CALCULATE(

[Cascade_Value_All],

DIM_Heading1[Heading1_Name]=”Sales”,

DIM_DataType[Data_Type_Name]=”Actual”,

DATESBETWEEN(

Dates[Date],

DATEADD(

LASTDATE(MA_Dates[Next_Month_Start_Date]),

MAX(MA_Function_Periods[Moving_Average_No_Periods])*-1,MONTH

   ),

LASTDATE(VALUES(MA_Dates[Month_End_Date]))

         )

  )

)

  )

The IF statement works as follows:

  1. I first need to determine that I’m evaluating only where I have one value for MA_Date[Month_End_Date].  If I don’t do this, I get that old favourite error in my subsequent evaluation that says that a table of multiple values was supplied……
  2. I then evaluate to determine if my column heading date (VALUES(MA_Dates[Month_End_Date]) is less than or equal to the date selected on the Month End Period slicer (LASTDATE(dates[Date_Month_End])…AND (&&)
  3. My column heading date is greater than or equal to a calculated date which is X periods prior to the selected “Show Periods Up To” as selected on the Slicer.  I use a DATEADD function for this similar to that used in my CALCULATE function except we’re adjusting the date by the value selected on the “Trend No of Periods” slicer.

With this in place, we have the total sales for the selected period relating to the users selections.

image

So my table is now limited to the number of trend periods selected and represents the month end date selected.

So Now We Just Divide By “Moving Average No of Periods” Right? eh NO!

We’ve calculated our total sales for the period relating to the users selections.  You would be forgiven for suggesting that we simply divide by the number of moving average periods selected.  Depending on your data, you could do this but the problem is that the dataset may not hold the selected number of periods, especially if the user can select a month end date that goes back in time.  As a result, we need to work out how may periods are present in our Sales_Moving_Average_Total_Value measure.

Sales_Moving_Average_Periods

= IF(COUNTROWS(VALUES(MA_Dates[Month_End_Date]))=1,

    IF(VALUES(MA_Dates[Month_End_Date])<=

LASTDATE(Dates[Date_Month_End])

&&VALUES(MA_Dates[Month_End_Date])>=

DATEADD(

LASTDATE(Dates[Date_Next_Month_Start]),

(MAX(MA_Trend_Periods[Trend_Periods])*-1),MONTH),

CALCULATE(

COUNTROWS(DISTINCT(FACT_Tran[Fact_Month_End_Date])),

DIM_Heading1[Heading1_Name]=”Sales”,

DIM_DataType[Data_Type_Name]=”Actual”,

DATESBETWEEN(

Dates[Date],

DATEADD(LASTDATE(MA_Dates[Next_Month_Start_Date]),

MAX(MA_Function_Periods[Moving_Average_No_Periods])*-1,MONTH),

LASTDATE(VALUES(MA_Dates[Month_End_Date]))

         )

   )

)

  )

This measure is essentially the same as my Sales_Moving_Average_Total measure.  The only real difference is that we count the distinct date values in our dataset as opposed to calling the Cascade_Value_All measure.  I mentioned earlier that there was a reason why I needed the month end date to be held on my FACT_Tran table and this is why.  If I use any other table holding the month end date, that table isn’t going to have been filtered in the way that the core dataset has been filtered.  As an example, my Dates table has a series of dates that spans my dataset timeframe and more.  As a result, evaluating against this table will deduce that the table does in fact have dates that precede my dataset and there is therefore no evaluation as to whether there is a transaction held in the dataset for that date.

image

As you can see, since my dataset runs from 1st July 2009, I only have 9 periods of data to evaluate for my 31/03/2010 column.  If I had divided by 12 (as per my “Moving Average No of Periods” slicer selection), I would have got a very wrong answer.  Obviously, this is slightly contrived but it’s worthy of consideration.

And Now The Simple Bit

I can understand that the last two measures have taken some absorbing, especially working out when particular date fields should be used.  For some light relief, the next measure won’t really tax you!

Sales_Moving_Average_Value =

IFERROR(

[Sales_Moving_Average_Total_Value]/[Sales_Moving_Average_Periods],

BLANK()

    )

This is a simple division with a bit of error checking to avoid any nasties.

image

When It’s All Put Together

Since all of these measure are portable, I can create another Pivot Table on the same basis as the one above (with Sales_Moving_Average_Value given an alias of Moving Average), move some stuff around, add a measure for the actual sales value for the month (I won’t go into that now, but it’s a simple CALCULATE measure with some time intelligence) and I then reconfigure to look like the following:

image

I can then drive a simple line chart and apply a trend line to my “Actual” measure with the chart conveniently hiding my data grid that drives it.

image

As you can see, a trend on my Actual measure shows a steady decline.  My Moving Average, however, shows a relatively stable, if not slightly improving trend.  Seasonality of some other spikes are obviously therefore involved and the reality is that both measures probably need to be reviewed side by side.

For those of you reading this who are interested in seeing the workbook of this example, I’ll look to post this in a future post when I take this analysis one step further to cover the whole P&L.  Sorry to make you wait.

I hope this helps you out BillD…

One More Point to Note

Those eagle eyed DAX pros out there have probably noticed that my IF functions only contain a calculation to evaluate when the logical test reaches a True answer.  The reason is that the function assumes BLANK() when a false evaluation condition isn’t provided.  I haven’t worked out if there’s any performance impact using this method on large datasets.  It’s up to you what you chose to do and if anyone can convince me why coding the False condition as BLANK() is best practice, I will quickly change my habits!


Profit & Loss (Part 3)–Return On Sales and Variances

October 27, 2011

 

P&L Graphic

Guest Post by David Churchward

PART 1 and PART 2 of this series on Profit and Loss posts covered the basic layout of the P&L together with some time intelligence and filtering to display relevant numbers to cover actual, budget and prior year for both a selected period and the equivalent year to date.  This was all based around the core measure referred to as Cascade_Value_All.

In this post, I’ll go on to refine this report further by adding Return On Sales (ROS) percentages and variance calculations as well as tune up the appearance of the report.

Return on Sales (ROS)

Return on Sales (or ROS for short) is the percentage that a number on our report represents as a function of the equivalent sales value.  Most people will be familiar with the term Gross Margin Percentage and this is one type of ROS.  Gross Margin percentage is a function of gross margin value divided by sales value.  This is often dissected by individual product lines as the gross margin values are analysed by the same product categories as sales.  Operating Profit Percentage is another ROS measurement but this is not normally sub analysed (although it can be if your dataset lends itself to that approach).

You’ve probably deduced that the financial calculation is therefore:

ROS %= Analysed Value / Equivalent Sales Value * 100

We’ve already got the value to be analysed in our model.  This is the Cascade_Value set of measures.  Therefore, we simply need to create the Equivalent Sales Value measure.  I’ve called this measure Sales_Compare.

Calculating the Equivalent Sales Value – Sales Compare

In PART 2, we ended up with 6 measures (Cascade_Month_Actual, Cascade_Month_Budget, Cascade_Month_Actual_PY and the equivalents for YTD).  We can use these measures as the basis for for a new set of Sales_Compare measures.  I’ll run this example on the Cascade_Month_Actual measure.  The same methodology applies for the other Sales Compare measures that we’ll use for each of our final report headings.

My Cascade_Month_Actual carefully sums the underlying dataset for each of my report headings (Report Heading 1 contains my main groups and Report Heading 2 contains my sub groups).  I need to maintain my Report Heading 2 categorisation but remove the link from my Report Heading 1 categorisation.  This is done using an ALL() function within a CALCULATE function.

SalesComp_Month_Actual_Interim

=CALCULATE(

[Cascade_Month_Actual],

ALL(DIM_Heading1),

DIM_Heading1[Heading1_Name]=”Sales”

       )

You’ll notice that I use my Cascade_Month_Actual measure, remove any association that it may have with Heading 1 and then direct it to filter the measure based on Heading 1 being equal to “Sales”.  This way, I essentially repeat my sales value in each section of my P&L report.

image

It should be noted that this is an interim calculation.  It doesn’t necessarily make sense as it stands in the report above, but it will prove useful in further calculations.  Having said that, I would prefer to only see the values in the sections where I want to display ROS percentages.  In order to do this, I need to tell my dataset where I want to see ROS percentages displayed.  To do this, I go to my DIM_Heading1 table and create a field where I can mark the headings where I want ROS calculations to be performed.  I’ve called this field Heading1_PCT and mark the headings where I want to drive this calculation with a 1.

image

I can now reference the Heading1_PCT field to determine when the Sales Compare measure should return a result.  We’ll call this measure SalesComp_Month_Actual.

SalesComp_Month_Actual

= IF(

COUNTROWS(VALUES(DIM_Heading1[Heading1_PCT]))=1,

IF(VALUES(DIM_Heading1[Heading1_PCT])=1,

CALCULATE(

[Cascade_Month_Actual],

ALL(DIM_Heading1),

DIM_Heading1[Heading1_Name]=”Sales”

       ),

           BLANK()

         ),

    BLANK()

    )

The calculate function is exactly as per our previous measure but I’m only running the calculate function when Heading1_PCT is equal to 1.  I can only conduct this evaluation when I only have 1 value for Heading1_PCT so I have to use COUNTROWS and VALUES to determine this.

image

Calculating ROS

We now have the two key measures to derive our ROS being Cascade_Month_Actual and SalesComp_Month_Actual.  I could simply divide one by the other surely.  Essentially, that is correct, but as explained at length in Profit & Loss – The Art of Cascading Subtotals, we need the measure to behave slightly differently in different sections of the report.  You’ll notice above that I have SalesComp_Month_Actual detail values for Operating Profit and Gross Margin.  I only want my ROS calculation to evaluate at the total level for Operating Profit but at the detail level for Gross Margin.  I can therefore use the evaluation that I used in my Cascade_Value_All measure and join it up with my simple division.

ROS_Month_Actual

= IF(

MAX(DIM_Heading1[Heading1_Summary]) = 1

&&(MAX(DIM_Heading1[Heading1_Show_Detail]) = 1

||COUNTROWS(VALUES(DIM_Heading2[Heading2_Name])) > 1

          ),

[Cascade_Month_Actual]/[SalesComp_Month_Actual],

    blank()

     )

My logical test is exactly the same as the one we saw in Cascade_Value_All in Profit & Loss – The Art of Cascading Subtotals.  After that, it’s a simply division.

image

This same method is used for each associated column measure that we wish to calculate an ROS for.  If I also remove my interim measures and apply some formatting, I get the layout below for month actual, month budget and month prior year.  I should take this opportunity to apologise for not having been very inventive when I made this dataset up having used a flat set of percentages!

image

Variances

To complete the picture, we need to add some variance calculations.  Because of the way that we’ve built up our measures, this is now very simple.  Our actual versus budget variance is

Variance_Month_AvB

= [Cascade_Month_Actual] – [Cascade_Month_Budget]

I’m not going to insult your intelligence by writing out the equivalent for actual versus prior year.

Tidy Up

My report is going to display data for a selected month and the year to date position for that month.  I prefer to split the two sections with a blank column.  I don’t think we’re going to register on Rob’s spicy scale for this one!

Blank = BLANK()

Headings and CUBEVALUE

To give my measure names meaning and to allow for their careful use, they’ve become somewhat ugly.  You can of course provide a title for the column that is a lot more user friendly.  However, I prefer to write my own because I can make it look tidier and, more importantly, I can reference the period that has been selected on my slicer.

I use an Excel formula in my heading that uses a CUBEVALUE function to call the period number that I’ve selected.  You could also do the same with year selections.

=”Period – “&CUBEVALUE(“PowerPivot Data”,”[Measures].[Maximum of Period]“,Slicer_Period)

This appends the period number onto the text “Period – “ by referencing the MAX value for Period when brought into the context of my slicer called Slicer_Period.  I do the same for YTD, add a few more headings to describe the type of data and then hide my pivot table headings!

When It All Comes Together

When you put all of these elements together (PART1, PART2, Sales Compare, ROS, Variances, Blank and Headings) you get something like this….

image

You can DOWNLOAD THE WORKBOOK HERE


Profit & Loss (Part 2)–Compare and Analyse

October 20, 2011

PandL Full Screenshot

Guest post by David Churchward

In my recent post, Profit & Loss-The Art of the Cascading Subtotals, I went through a basic P&L layout with some relatively complex DAX measures to display and hide row headings as appropriate together with calculating accurate values.  In order to make this report more meaningful, it needs comparatives and further analysis.  In this post, I’ll build on the P&L created in part 1 to create some of the key elements of the layout shown above including Actual values, Budget values and Prior Year values for a selected period and the associated year to date (YTD).  In part 3, I’ll go on to show how the percentage calculations work and maybe some pointers for making it look REALLY good!

Time Intelligence

As our report is considering different timeframes, we need to establish a time dimension and condition slicers to select a required timeframe.  From this, we can determine the required time parameters for use in subsequent measures.

We need to establish 4 tables that we will use for time intelligence.  These are fully explained in my recent post Slicers for Selecting Last “X” Periods.

Dates – this is a list of sequential dates covering the timespan of our dataset.  This is linked to a date field in the dataset.

Year – this is a sequential list of years covering the timespan of the dataset.  This table should NOT be linked to the core dataset (or fact table if you prefer).

Period – this is a sequential list of periods, normally from 1 to 12 where a period is a calendar or fiscal month.  Again, this table should NOT be linked to the core dataset.

Year_Period – this is a list of year and period combinations covering the timeframe of the dataset.  This table also carries other relevant dates and attributes that relate to the date records contained within.

Year_Period is linked to both Year and Period on a many to one basis.

Creating Time Parameters

I create time parameter measures on the Year_Period table.  We need the following time parameters for use in our measures:

Selected Month End Date – this tells us the date relating to the end of the month for the selected period and year combination selected by the user on slicers.  Month end dates don’t change and so I’ve been able to hold this as a field in the Year_Period table which means that I just need to capture the associated value and deal with the fact that multiple selections could be made (or no selection at all).  This is done using:

Selected_Month_End_Date = LASTDATE(Year_Period[Month_End_Date])

It should be noted that I’m always using the LASTDATE function in these measures.  This is to ensure that I always evaluate to one result.

Selected Month Start Date – again, this is available in the Year_Period table as the value doesn’t change for any selected date

Selected_Month_Start_Date = LASTDATE(Year_Period[Month_Start_Date])

Selected Prior Year Month End Date – once again, we could make this available on the Year_Period table as the value won’t change and, for efficiency reasons, I would tend to do that.  However, for the benefit of showing an additional method, I’ve chosen to use the DATEADD function here.

Selected_PY_Month_End_Date = LASTDATE(DATEADD(Year_Period[Month_End_Date],-1,YEAR))

DATEADD requires the syntax DATEADD(dates, number of intervals, interval).  The dates element is a table / column expression which details which column to use in the evaluation.  I use –1 as an interval to go back in time (essentially turning DATEADD into DATEMINUS which of course doesn’t exist as a function!).  The interval is the timeframe type by which you wish to adjust and this can be DAY, MONTH or YEAR as my date field doesn’t contain any time elements.

Selected Prior Year Month Start Date – let’s not let this drag out and get boring!  Measure below:

Selected_PY_Month_Start_Date = LASTDATE(DATEADD(Year_Period[Month_Start_Date],-1,YEAR))

Selected Year Start Date – DATEADD is used again here but we need to know how many months to go back.  As the user is selecting a period and year combination, we can pick up the period that is being evaluated from the slicer and use this to work back to the first date in the year.

Selected_Year_Start_Date = LASTDATE(DATEADD(Year_Period[Next_Month_Start_Date],MAX(Year_Period[Fiscal_Period])*-1,MONTH))

As I’m using Fiscal Periods (in the main I use July as Period 1), I need to pick up the selected Fiscal Period (ensuring that I evaluate to one answer – hence the MAX function) and then multiply by –1 to work backwards through time as opposed to forwards.

Selected Prior Year Start Date – this is getting boring Churchy – move on:

Selected_PY_Year_Start_Date = LASTDATE(DATEADD(DATEADD(Year_Period[Next_Month_Start_Date],MAX(Year_Period[Fiscal_Period])*-1,MONTH),-1,YEAR))

Nothing tricky here.  I just take one year away from Selected_Year_Start_Date.  I’ve used a nested DATEADD to show how it’s done as opposed to using the answer from Selected_Year_Start_Date.  There’s multiple ways of doing these things!

With that done, I think we’re finished with date parameters.  Let’s get on with putting them to good use.

Applying Time Intelligence to my Cascade_Value_All Measure

You may recall in Profit & Loss-The Art of the Cascading Subtotals that the key outcome was a measure called Cascade_Value_All.  We can now start dissecting this measure by overlaying time intelligence and filters to capture the values that we need on our report and this is all done safe in the knowledge that Cascade_Value_All will manfully ensure that report headings behave appropriately.

I’ll construct these measures in sections to make them clear.  You can ultimately combine these into fewer measures if required.  Let’s start by overlaying time intelligence.  This is done using the DATESBETWEEN function.  I tend to use this regularly as I’ve never encountered a situation where I’m working with calendar years!

We need 4 measures – current month, current month last year, current month YTD, current month YTD last year.

Cascade_Month =

CALCULATE(

[Cascade_Value_All],

DATESBETWEEN(

Dates[Date],

Year_Period[Selected_Month_Start_Date],

Year_Period[Selected_Month_End_Date]

         )

  )

In this measure, we’re essentially filtering the outcome of Cascade_Value_All down to the underlying transactions that fit between the start and end dates provided to the measure.  This means that our other 3 required measures that represent current month last year, current month YTD and current month YTD last year are exactly the same although the start_date and end_date elements highlighted in bold are substituted with the relevant date parameters calculated in the Creating Time Parameters section above.  Let’s call these three measures Cascade_Month_PY, Cascade_YTD and Cascade_YTD_PY.  Don’t worry as all measures will be available in the workbook that I’ll make available with Part 3.

Filter Actual and Budget

In our P&L report, we have two types of data being Actual and Budget.  These are all records in the main fact table dataset and each record is denoted with a Data_Type field (value of 1 denoting Actual and a value of 2 denoting Budget).  This field is linked to a DIM_DataType table.

image

To filter the dataset for the current month measure (I won’t go through them all as I’m sure you’ll get the idea), I use the following measure:

Cascade_Month_Actual =

CALCULATE(

[Cascade_Month],

DIM_DataType[Data_Type_Name]=”Actual

      )

This will give us the correct values for the selected month representing the “Actual” dataset.  We can create an equivalent for “Budget” simply by changing the filter value.

I create a “Prior Year” version of Cascade_Month_Actual by substituting [Cascade_Month] with [Cascade_Month_PY], ensuring that our filter is set to “Actual”.

Why Filter “Actual” and “Budget” instead of simply adding them to the Column Headings in the Pivot Table? – Use of Static Columns

I could avoid multiple measures by simply using my [Cascade_Month] measure (together with the other time adjusted measures) and adding Data_Type as a column heading.  There’s a few reasons why I wouldn’t do this in this particular case.  That’s not to say that it isn’t a valid approach in the majority of other examples.  My reasons are:

  1. I want to ensure that my columns remain static.  If, for whatever reason, I evaluate to not having a budget for a particular slice of the data, I don’t want the column to disappear.
  2. OK, so number 1 might be a bit weak!  I’m also aware that the budget for Prior Year probably isn’t relevant so I don’t want to see it.  My prior year comparison is being used to evaluate my current year actual.  I don’t want prior year budget getting in the way so I only want to see actual for my prior year dataset.
  3. I want to add a blank column between my “Month” information and my “YTD” information.  Again, a bit weak, but buy-in from users is partly about how pretty is looks and I think it looks prettier this way.
  4. If I use a field on my column headings, I could get multiple rows dedicated to my column titles.  As a taster to what’s coming up, I don’t want to use the Pivot Table headings because they’re ugly on this report.  I’m going to create my own damn it!  If I use static columns, I can guarantee only one heading row which means I can hide it and create my own prettier version!
  5. This is probably the most compelling reason!  It’s quite normal for Budgets to be superseded by Revised Forecasts.  With static columns, I can ultimately provide a slicer that allows the user to select whether they want to see budget or revised forecast information against actuals.

Note – there’s always a balance to be considered when making an assessment like this.  On the one hand, this may “look pretty” and avoid displaying information I don’t want but if performance suffers than you’re onto a rough deal!  With these measures, I haven’t had a performance issue with quite large datasets so I don’t have to compromise appearance just yet but there’s no doubt that it’s less optimised than constructing one measure and adding a field to column headings.

When We Pull This All Together……

I add the six new measures to the P&L report and attach slicers for Period [Period] and Year [Year], tidy up a bit and produce the following:

PandL Post 2 Screenshot

Look out for Part 3 when we’ll add ROS (Return on Sales) percentages and variance calculations whilst also having a tidy up of the layout.


Slicers For Selecting Last "X" Periods

October 11, 2011

 

Guest post by David Churchward

Using Time Intelligence in PowerPivot can appear scary when you first start using it and I’ve seen some weird and wonderful ways of attacking it, some that look like we’re calling into question the validity of the global phenomenon that is time and others that are quite simply brilliant.

In this post, I hope to provide a simple and understandable approach whilst adding another flexible dimension called “X” where “X” is a variable that can be set by a user in a slicer.  This might be, for example, a P&L report where the user wants to see the last 4 months instead of the normal pre-determined timeframes such as year to date or quarters.  Alternatively, the report may be reviewing trends and, again, year to date or one of the other standard timeframes may not be the most relevant timeframe.

Incidentally – if you’re in the mood for conquering time intelligence there’s a host of brilliant information on this site including posts such as Running Totals Without a Traditional Calendar Table and PowerPivot time intelligent functions: why use ALL() and how to work around it which give some excellent direction on Time Intelligence and some of the complexities.

In this post, I’ll explain how you can put all of the control in the hands of the user by creating an X Periods measure. That is to say that the user can review the last X periods up to a date that they control.

Paving the Way

The crux of dealing with time intelligence in this way is to create a time table that links to your dataset and then a further time selection set of tables from which the user can select their required time parameters.

I’ve created so many of these where the fiscal year isn’t a conventional calendar year that I’ve found that it’s best to deal with the fact that some of the built-in time intelligence PowerPivot functions may not be valid. However, with last X periods, it doesn’t matter anyway.

I need the following tables in my dataset:

FACT_Tran- this is my fact table dataset that I wish to analyse.

clip_image002

Dates- this is an unbroken list of dates representing the timeframe of my dataset. Time Intelligence works better when your fact table is linked to a dates table, chronologically ordered without any breaks.

clip_image002

Year- a list of years in the dataset

clip_image003

Period - a list of periods in the dataset. This would normally be 1 to 12

clip_image004

Year_Period- this is more aligned to a proper time dimension in that it is a combination of the years and periods representing my dataset and carries additional information such as month end dates that we need for our measures (this version has been simplified for the purposes of this post)

clip_image005

X- this is a single column table that allows users to select the number of periods that they wish to see. This could be a numeric sequential number or specific timeframes that are most likely to be selected as in my example here.

clip_image006

Note – I’m using a predetermined set of values for X which contradicts my opening remarks but this is purely for the purposes of showing this example and it really depends on the situation!

Links

clip_image007

You’ll notice that my Year_Period table is linked to my Year and Period tables. My fact table is separately linked to my Dates table. For this method of Time Intelligence to work, you shouldn’t link the two sets of tables together otherwise filtering will occur when we don’t want it to.  You can get around this, but there’s no need to link the two sets of tables anyway.

Determining what has been Selected

Users will be given slicers to select the year and period that they are working on. In terms of X periods, we would consider this to drive the end date of the period to which X relates.  We then work back X periods from that date.

I need to create a series of measures to determine what the user has selected. These are as follows:

Selected_Month_End_Date = LASTDATE(Year_Period[Month_End_Date])

Selected_X = MAX(X[X])

Selected_X_Months_Start_Date = LASTDATE(DATEADD(Year_Period[Next_Month_Start_Date],[Selected_X]*-1,MONTH))

clip_image008

As I’m using Fiscal Periods and my financial year starts in July, period 3 for the year ending 2010 (as selected above) is actually September 2009. I’ve selected 9 as a value for X which means that I’m looking back 9 months from September 2009. My start date is therefore 1st Jan 2009.

Building These Dates into my Financial Measure

Now that we know what the user has selected in terms of X and the date range to which X relates, our financial measure is now very simple.

X_Periods_Value

=CALCULATE

(

SUM(FACT_Tran[Value]),

DATESBETWEEN(Dates[Date],[Selected_X_Months_Start_Date],[Selected_Month_End_Date])

)

We now have a dynamic measure which calculates the start and end dates that we wish to see and applies those dates to our value column.

This can be massaged to calculate year to date, prior year and many other date values as required and it is totally dynamic without any back end calculations.

Taking Things to the Next Level

The method above is what I use regularly to allow for year, period and X selectors to drive the analysis.  This works a treat if your primary reporting timeframe is a month.  However, what if days or weeks are a relevant timeframe?

Let’s adapt what we’ve already done by creating a new dates table.  Let’s call this table Dates2.  This is a sequential date column similar (if not identical) to Dates but NOT linked to our fact table.  Create 3 new measures on this table which are variations of the Selected_X_Months_Start_Date measure that we created previously.

Selected_X_Days_Start_Date

= DATEADD(LASTDATE(DATEADD(Dates2[Date],[Selected_X]*-1,DAY)),1,DAY)

 

Selected_X_Weeks_Start_Date

= DATEADD(LASTDATE(DATEADD(Dates2[Date],[Selected_X]*-7,DAY)),1,DAY)

 

Selected_X_Months_Start_Date

= DATEADD(LASTDATE(DATEADD(Dates2[Date],[Selected_X]*-1,MONTH)),1,DAY)

So why do we need two DATEADD functions in each of these measures? Simply put, we’re essentially using dates here. That’s a simple statement that doesn’t answer anything! What I mean is that we’re essentially not using datetime fields. Each date is therefore a value in it’s own right. Therefore, if we treat 30th September as a day only and therefore a value of 30, when we subtract one day, we would get 29. As a result, we would be reviewing 29th and 30th when in reality we only want to review one day. Since 30th is one day in it’s own right, we always have to add one day to each answer.

Note – take care to ensure that your fact table dates are in the same format.  Time can skew the answer if you’re not careful.

In the previous solution using year and period slicers to select months above, this problem was dealt with by using the Next_Month_Start_Date value which is one day ahead of our month end date selected and held as a value in our Year_Period table.

Our weeks solution is a derivative of the days function where we simply adjust by days * 7 to get to weeks.

I now create another selection table called Time_Type.  This is designed to select which timeframe type the user is selecting.

image

I create a slicer on my pivot table using the Type_Name and create a measure which determines which item has been selected by the user.

Selected_TimeType

= MAX(TimeType[Type_Code])

I then create another measure which uses the selected time type to determine which of the 3 measures to use:

Selected_X_TimeType_Start_Date

= IF([Selected_TimeType]=1,[Selected_X_Days_Start_Date],

IF([Selected_TimeType]=2,[Selected_X_Weeks_Start_Date],

IF([Selected_TimeType]=3,[Selected_X_Months_Start_Date],

   BLANK()

  )

   )

     )

 

image

We need one further measure to determine which date has been selected

Selected_Date2

= LASTDATE(Dates2[Date])

These new measures can now be used in our value measure:

X_Periods_Value

=CALCULATE

(

SUM(FACT_Tran[Value]),

DATESBETWEEN(Dates[Date],[Selected_X_TimeType_Start_Date],[Selected_Date2])

)

What if I Always Want my Report to be X Periods to Today’s Date?

This is simple.  We simply need to substitute Selected_Date2 for a new measure called Todays_Date in our X_Periods_Value measure and substitute Dates2[Date] with Todays_Date in Selected_X_Days_Start_Date and Selected_X_Weeks_Start_Date and Selected_X_Months_Start_Date measures

Todays_Date

= TODAY()

You do however need to take care about what your data content is.  As an example, I use a data warehouse that is built every night.  When I refer to today, I actually mean last night which means that my measure should in fact be

Todays_Date_LastNight

= TODAY() -1

What Does This Mean?

By using this method, we have passed the requirement to determine the timeframe to which a trend or report relates across to the user.  Trend timeframes could change and do we really want to have to recreate reports and dashboards as a result?  I hope that this solution solves that problem.

Having learnt from my previous post, YOU CAN DOWNLOAD THE WORKBOOK HERE.