Friday Bonus: Gang Sign Photos So Far

December 8, 2011

 

This is my favorite so far, by the narrowest of margins.  It comes from Angel Abundez of RDP Streamline Solutions.  And Angel (twitter) executes quite well.  Note the form.  Good X – not the V or the Y that most people’s X’s are turning out to be.  And a sharp L.  The background pic is nice too.

But his daughter simply nails it.  Look at that total lack of emotion.  A stone cold spreadsheet killa.  Everyone should aspire to her standard of excellence.

VIEW THE FULL GALLERY HERE

I’m gonna leave submission open for a few more days.  Plenty of time to get yours in!

SUBMIT YOUR PICTURE(S) HERE


The Greatest Formula in the World, Part Two

December 7, 2011

image

Our Time Machine From Part 1 Needs a Couple of Fixes

OK, if you recall from part one, we had a “year over year” time machine calculation going on, and it was built against a data set that lacked a “real” calendar table.  Very cool.  Very resourceful.  And very necessary.

But the time machine had two problems.  Let’s fix them.

Problem #1:  Meaningless Grand Total

Let’s start with the easy one:

Year over year in PowerPivot - Meaningless Grand Total

Grand Total is Meaningless for Last Year Sales

It is meaningless to have a “grand total” value for a measure that returns “last year’s sales.”  What year would that BE, actually?  It’s nonsense.  So we use an old trick, one of the many flavors of IF(VALUES()):

=IF(COUNTROWS(VALUES(Periods[Year]))>1,BLANK(),
   CALCULATE([Total Sales],
      ALL(Periods),
      FILTER(ALL(Periods), Periods[Year]=MAX(Periods[Year])-1),
      VALUES(Periods[MerchPeriod])
   )
)

Where the new IF “wrapper” is highlighted and the original formula is in normal font.

For more on that “IF VALUES” technique, please see this post.

The results are as desired, the grand total is now blank:

image

Problem solved.  Moving on…

Problem #2:  2011 isn’t complete but we’re getting all of 2010 sales

Remember, our sales data for 2011 only goes through the first 6 months.  So we do NOT want “last year sales” for months 7-12:

image

That can be solved a number of ways.  The simplest is just to add another IF(), and have the Last Year Sales measure return BLANK() whenever there are no Sales:

=IF(COUNTROWS(VALUES(Periods[Year]))>1,BLANK(),
   IF([Total Sales] = BLANK(), BLANK(),
      CALCULATE([Total Sales],
         ALL(Periods),
         FILTER(ALL(Periods),
            Periods[Year]=MAX(Periods[Year])-1
         ),
         VALUES(Periods[MerchPeriod])
      )
  )
)

That trims the results to just the months desired:

image

But note that the subtotal for 2011 is still too high:

image

“Last Year Sales” for 2011 Should Only be Returning
2010 Sales Through the First Six Months

This one is trickier.  I struggled to find a good answer before coming up with the following:

Step 1:  Add a “Next Year Period Num” Column to the Periods Table

Recall that my Sales table has a “Period Num” column that is the basis for the relationship with the Periods table:

image

And that matches up with a similar column in the Periods table:

image

Note that Period Num does NOT reset to 1 with each new year (unlike MerchPeriod).  So that is the absolute unique ID for a given Month/Year combo.

Well, I created a new column in the Periods table that tells me, for a given Period Num, what the equivalent Period Num will be NEXT year:

image

Now, in my FILTER statements, I can choose to match on THAT column instead.

Bringing it Home

OK, now that I have that column, I can add a new FILTER clause to my measure:

=IF(COUNTROWS(VALUES(Periods[Year]))>1,BLANK(),
   CALCULATE([Total Sales],
      ALL(Periods),
      FILTER(ALL(Periods),
         Periods[Year]=MAX(Periods[Year])-1
      ),
      FILTER(ALL(Periods),
         Periods[NextYear Period]<=
            MAXX(
                VALUES(PeriodSales[Period Num]),
                PeriodSales[Period Num]
                )
      ),
       
      VALUES(Periods[MerchPeriod])
   )
)

OK, what does that do?  Simply put, it further filters the Periods table to NOT go beyond any periods for which we currently have records in the Sales table.

And the results:

Last Year Sales Measure in PowerPivot With a Custom Calendar

Last Year Sales Measure Doing the Right Thing

And now if you want YOY Growth, it’s straightforward:

=([Total Sales] – [LY Sales Finished]) / [Total Sales]

Which yields:

Year over Year / Year on Year Growth Percentage Measure in PowerPivot With a Custom Calendar

Year over Year / Year on Year Growth Percentage
Measure in PowerPivot With a Custom Calendar

OK, I cheated and wrapped another IF(COUNTROWS(VALUES)) around the outside of the formula so that the grand total cell is blank again, because again, that’s a meaningless value to report.  Details.

Isn’t that a lot of work?

Depends on how you look at it.  First of all, it’s a lot harder to grasp the first time than it is as you get used to it, trust me.

But even better, this investment pays off forever.  Normal Excel formulas may be easier to write the first time, but then you have to re-write and adjust them forever, every time your data changes.  And every time your desired report shape changes.

This one is a portable formula.  It goes wherever you want it to go, and eats whatever data you feed it.  Forever.  It’s worth a little extra effort.

OK but do I really have to understand all of that?

No, not really.  Not right away.  Think of it this way:  this is what I do all day, every day.  And I can tell you that figuring this out is not something you want to do in your spare time.

But once someone gives you the pattern?  Wash, rinse, repeat.  Make small adjustments as needed.

Honestly, that whole formula above could be wrapped up in a function provided by Microsoft, and all of the nasty stuff hidden.  I told them as much last time I visited Redmond.  And if it were a function rather than a formula, we wouldn’t care at all how it worked.

So… treat this as a pattern.  Copy/paste and modify to fit your needs.  Seriously.

Unfinished Business

I actually WILL retrace my steps here though and explain a number of things though, so that you CAN understand.  I have a couple of posts in the queue aimed at just that.

Were there alternate ways to write this measure?

Yes, there were, especially in that last step where I needed to get the “Last year sales” measure correct for the 2011 subtotal.  I could have used SUMX to make the year subtotal equal to the sum of its underlying months.  That is worthy of a post for sure.

But I was wondering if there was still another way.  A way that doesn’t use SUMX and doesn’t require a new calc column.

And for that, we need the Italians.  So I’m gonna try out my new toy, the Boot Signal:

image

Calling Marco and Alberto

Get it?  *Boot* Signal?


The Greatest Formula in the World, Part One

December 6, 2011

 
image

Play Write the greatest song formula in the world, or I’ll eat your souls.”
-From “Tribute,” by Tenacious DAX

A Do-It-Yourself Time Machine

You remember the Great Precedence Project?  Well, I have shelved it for now, for two reasons:

  1. Creating an exhaustive list of all the precedence rules in PowerPivot formulas was going to be…  exhausting.
  2. I didn’t really need to boil the philosophical ocean in search of deep underlying meaning, because on further inspection, I already had the Greatest Formula in the World, and needed no other.

So what is the GFITW about?  It’s about time navigation.  It’s useful when you have a custom calendar and the time intelligence functions are therefore not so helpful.

The GFITW is a do-it-yourself time machine.

Year on Year Sales With a Custom Calendar?  Yes We Can!

In that post linked above, which happened to be the last installment of the precedence project, I was calculating a running total, and doing so without benefit of the fancier functions like DATESYTD etc. (which rely on the “real” calendar, not your business calendar).

OK, so now let’s do a “Year on Year” sales measure, sometimes called a “Year over Year” measure.  And again, our “calendar” table isn’t a true calendar at all:

Semantically this IS a calendar but functions like PREVIOUSYEAR can't use it

A Table of Periods (Not Dates) Means You Cannot Use the Built-In
Time Intelligence Functions to Calculate Year-on-Year Sales

And the Sales table is very simple:

Sales Table linked to a period number, not linked to a date

Sales Table is Linked to the Periods Table by the Period Num Column

And I have this basic pivot already set up:

image

Year and MerchPeriod on Rows

So, how do I write a measure that “fetches” the sales from last year?

Let’s Skip Some Steps

I am going to be honest with you:  it took me a very long time to figure this out.  The formula looks reasonably simple, and it is, but I had to try a million variations before I got it right.  Literally, it took me about a week to refine it.

So rather than walk you through that painful process, let’s skip ahead and show you the GFITW in all its glory.  Then in the next post I can show you what NOT to do, and we can also fine tune it.  OK?  OK.

So here it is, the Greatest Formula.  In the World:

=CALCULATE([Total Sales],
   ALL(Periods),
   FILTER(ALL(Periods),
      Periods[Year]=MAX(Periods[Year])-1),
   VALUES(Periods[MerchPeriod])
)

And its results:

image

No, it’s not perfect yet.  We will need to calculate percentage growth of course, rather than merely fetching last year’s sales.  And that given that 2011 is not yet complete, I’m a little squeamish about Last Year Sales returning ALL of 2010’s sales (the $5.9M number).

But for now, let’s count our blessings.  This formula DOES fetch the sales from last year, and it works (mostly) both at the year level and the MerchPeriod (aka Month) level.

Dissecting the Formula

Here is the formula again:

=CALCULATE([Total Sales],
   ALL(Periods),
   FILTER(ALL(Periods),
      Periods[Year]=MAX(Periods[Year])-1),
   VALUES(Periods[MerchPeriod])
)

But really, the GFITW is not a formula.  It is a pattern that you can re-use and modify.  So let’s look at it that way:

=CALCULATE(Original Measure,

   ALL(Your Custom Calendar Table),
  
   FILTER(ALL(Your Custom Calendar Table),
      Year Column Or Similar =
        
Expression that “moves” the Year Column
  
),
  
   VALUES(
Time Column That is More Granular
          Than Year, Like Month or MerchPeriod

  
)

)

Where everything in blue is “fixed” as part of the pattern, and everything in italics is something you can change.

Like I said, this is running a bit long for one post.  Come back Thursday for the rest Smile


Friday Bonus: Excel Goes Gangsta!

December 2, 2011

 
image

“You Better Step Outa My Grid”
Dick Moffat Helps Me Debut the “Official” Excel (XL) Gang Sign

I Think We Can All Agree This Was Long Overdue

See a need, fill a need.  That’s kinda my credo.  And we clearly needed a gang sign that represents the worldwide cartel of Excel Ninjas.  (And their innocuous front group, the Excel Monkeys).

OK maybe that wasn’t it.  Maybe I’ve just felt left out of the whole long-running fad of posting ridiculous pictures of yourself on Facebook, flashing gang signs (a practice that IMO should rank much higher than #6 on the list of 15 dumbest Facebook picture types, but it does get properly skewered in this music video).

OK that wasn’t it either.  Really, I just think this picture, and this IDEA, couldn’t POSSIBLY be any funnier.

An Opportunity for Ridiculous Irony

That fine example of American leadership, Rahm Emanuel, is purported to have said “never waste a crisis.”  Similarly, I like to say, “never waste a visit from Dick Moffat.”  We recently met up to swap DAX techniques and war stories, and I knew we had to do a picture.

“What would be most ridiculous?” was naturally my first thought, and what emerged was, in my opinion, some of my finest work:  the XL gang sign.

I mean, who could be a better cohort for this than a pacifist Canadian grandfather, Access/Excel MVP, and author of a top-5 all-time popular post on… wait for it…  cube formulas?  (That post is 1.5 years old and STILL gets 50 views a day!)

I mean, the most “gangsta” Dick has ever looked was back when he used to do oatmeal ads:

image

A Bold Move:  The Gang Sign Goes Open Source?

When discussing this with our COO (Frank Fulton), he pointed out that this was not something we could responsibly exploit for our own benefit, and that we were obligated to open source it.

It took some time for me to warm to the idea.  I mean, while still basking in the glow of invention, it’s hard to change gears and immediately just give it all away like that.  I mean, at least let me protect it with the GNU Public License or something.

But I think it’s for the best.  Precise licensing terms will be announced shortly, once our massive legal team has had time to iron out a few issues with the Federal Court of Appeals.

Show Dick How it is Done – Submit Your XL Gang Sign Pics!

If you look again at that picture up top, it’s clear that one of us took to this task a little more naturally than the other.  I need your help.

Send me your pics of you (and maybe your spreadsheet crew) flashing the sign.  I’ll post the best ones next week and maybe even throw in a few small prizes.

Send pics to:

outamygrid@pivotstream.com 

Some suggestions:

  1. Look “hard.”  Sell it.  Scowl.  Look unfriendly but don’t overdo it and get snarly.  You’re not 1980’s Hulk Hogan.  You gotta have that cool indifferent look of badness, as modeled here by a more recent Hulk Hogan.
  2. Get Creative with Your Irony. Standing in front of a bank of huge monitors on Wall Street would work.  A massive cubicle farm, also good.  Wearing an Apple t-shirt, whatever – just give it something.  I mean, I accessorized with Dick Moffat.  That sets a high bar right there folks.
  3. Pay attention to details of the sign itself.  Remember, you have to make it backwards for you so that others see it the right way.  Left hand is the “L.”  Thumb points out (to your left).  Right hand is the X.  Index finger is closest to you and points to your right.  Middle finger crosses it to your left.  Or take advantage of the open source licensing and modify it to your liking.
  4. Channel your long-simmering feelings of disrespect.  You are the exploited, ignored backbone of the international business world.  Seriously, you are.  The entire “BI” industry claims to do what is 99.99% done by YOU all day, every day.  Channel that rage.  Oh yeah, you know where this is going…
  5. Now is your time!  Remember, PowerPivot truly is a revolution.  Our revolution.  YOUR revolution.  Anyone who tells you that Excel will never “be real BI” or tries to pigeonhole it as “BI for small teams only” doesn’t yet understand how much PowerPivot has changed the rules.  Put THAT thought in your head… 

…and then go take a ridiculously funny picture.


Using Query Editor to Create a Time Table

December 1, 2011

 

Guest Post by David Churchward with Technical Input from Scott Senkeresty

The first thing I ever do when I create a new PowerPivot workbook is create the date tables that I need.  Until now, I’ve used a preconditioned SQL table to draw this information from.  Now, thanks to Scott Senkeresty, you can now create your own – from PowerPivot!

Scott is a true genius.  He spends a lot of time under the hood of Pivotstream operations working his magic and wizardry and, every now and then, he pops up for air to deliver a masterpiece like this.

 

It’s as Simple as Copy and Paste

If you read my last post SQL UNPIVOT Makes My Data Skinny, you’ll know where query editor is and the sort of tricks that it opens up within PowerPivot.  In this post, there’s some SQL scripts that create time tables and you can simply cut and paste them into the query editor and use them straight away.

Having said this, it’s worth checking out Rob’s post The Ultimate Date Table to see why a date table is important and where you can get one of your own from Azure DataMarket.  However, I think we’ve gone one step better now and given you a script that you can parameterise easily, condition relatively easily and setup extremely easily.  Add to that the fact that it runs in mere seconds!

AND there’s one more bonus, especially for Accountants, you can set this up for YOUR Fiscal Year!

Copy, Paste, Save – Simples!

To get going, simply copy and paste the script from THIS ATTACHMENT into query editor.  When you save it – bam – time table complete.

DECLARE @FirstDate DATE

DECLARE @LastDate DATE

SET @FirstDate =’1 Jan 2008′– Enter first date of calendar as ‘dd mmm yyyy’

SET @LastDate =’31 Dec 2015′– Enter last date of calendar as ‘dd mmm yyyy’

 

IF @FirstDate ISNULLOR @FirstDate =

                BEGIN

                                SET @FirstDate =’1 Jan 2000′

                END

               

IF @LastDate ISNULLOR @LastDate =

                OR @LastDate < @FirstDate

                ORDATEDIFF(DAY, @FirstDate, @LastDate)> 10000

                BEGIN

                                SET @LastDate =DATEADD(DAY, 10000, @FirstDate)

                END

 

                ;WITH CTE_DatesTable

                AS

                (

                  SELECT @FirstDate AS [Date]

         UNIONALL

         SELECTDATEADD(DAY, 1, [Date])

         FROM CTE_DatesTable

         WHEREDATEADD(DAY, 1, [Date])<= @LastDate

                )

 

                SELECT

                                Date,

                                DATEPART(DAYOFYEAR,Date)                             ASDayOfYear,

                                DATEPART(DAY,Date)                                          ASDay,

                                DATEPART(MONTH,Date)                                    ASMonth,

                                DATEPART(YEAR,Date)                                        ASYear,

                                DATENAME(WEEKDAY,Date)                               ASDayOfWeek,

                                LEFT(DATENAME(WEEKDAY,Date),3)                 AS DayOfWeekAbr,

                                DATEPART(WEEK,Date)                                       AS WeekOfYear,

                                DATEPART(QUARTER,Date)                                 ASQuarter,

                                DATENAME(MONTH,Date)                                  ASMonthName,

                                LEFT(DATENAME(MM,Date),3)                           AS MonthNameAbr,

                                DATEADD(DAY,-1,DATEADD(MONTH, 1,DATEADD(DAY,(DATEPART(DAY,

DATE)*-1)+1,Date)))                         AS MonthEndDate

 

                FROM

                CTE_DatesTable

 

                OPTION (MAXRECURSION 10000)

Without doing anything to this script, you’ll get a time table that spans Jan 2008 to Dec 2015.  That could be enough.  In which case, you’re done.  However, I’m sure you want more.

Change Date Parameters to Widen the Timeframe

The great news with this script is that it’s so easy to adjust to cover the timeframe that you need.  You just need to follow these simple couple of steps:

  1. Towards the top of the script you’ll see “SET @FirstDate = “.  You simply need to change the value to the first date in your dataset (or perhaps an earlier date to be sure).
  2. I’m sure you’ve already guessed step 2 – change “SET @LastDate = “ to a value that is beyond the last date in your dataset.

I almost wish I could make it more difficult but Scott’s nailed this so well that it’s almost impossible to get wrong – well almost!

A Quick Health Warning

This script uses a recursive CTE (Common Table Expression).  If you’re not careful, it’s very possible to spin this baby into an infinite cycle.  Without going into why this happens, here’s a couple of points to make sure that you don’t fall into that trap:

  1. The IF – BEGIN – SET – END statements are there to ensure that valid entries are made for @FirstDate and @LastDate.  It’s probably best to leave these in.
  2. At the end of the script, there’s a command OPTION(MAXRECURSION 10000).  This ensures that the script won’t loop more than 10,000 times (ie 10,000 days).  If you set the recursion to zero, this removes all limits and then you could get into an infinite cycle.

Don’t be scared to give it a go.  There’s controls here to stop anything going wrong and what’s the worst that could happen anyway!

The Next Level – Fiscal Elements

In all my time using SQL and PowerPivot, I’m yet to work in an environment where a year runs from 1st January to 31st December.  As a result, we refer to a Fiscal Year to represent this alternate calendar.  I took the liberty of hacking Scott’s script to add these elements in.

DECLARE @FirstDate DATE

DECLARE @LastDate DATE

DECLARE @FiscalStart INT

SET @FirstDate =’1 Jan 2008′– Enter first date of calendar as ‘dd mmm yyyy’

SET @LastDate =’31 Dec 2015′– Enter last date of calendar as ‘dd mmm yyyy’

SET @FiscalStart = 4 – First calendar period of Fical Year – 1=Jan, 2=Feb…., 12=Dec

 

IF @FirstDate ISNULLOR @FirstDate =

                BEGIN

                                SET @FirstDate =’1 Jan 2000′

                END

               

IF @LastDate ISNULLOR @LastDate =

                OR @LastDate < @FirstDate

                ORDATEDIFF(DAY, @FirstDate, @LastDate)> 10000

                BEGIN

                                SET @LastDate =DATEADD(DAY, 10000, @FirstDate)

                END

 

                ;WITH CTE_DatesTable

                AS

                (

                  SELECT @FirstDate AS [Date]

                 UNIONALL

                 SELECTDATEADD(DAY, 1, [Date])

                 FROM CTE_DatesTable

                 WHEREDATEADD(DAY, 1, [Date])<= @LastDate

                )

 

                SELECT

                                Date,

                                DATEPART(DAYOFYEAR,Date)                             ASDayOfYear,

                                DATEPART(DAY,Date)                                          ASDay,

                                DATEPART(MONTH,Date)                                    ASMonth,

                                DATEPART(YEAR,Date)                                        ASYear,

                                DATENAME(WEEKDAY,Date)                               ASDayOfWeek,

                                LEFT(DATENAME(WEEKDAY,Date),3)                 AS DayOfWeekAbr,

                                DATEPART(WEEK,Date)                                       AS WeekOfYear,

                                DATEPART(QUARTER,Date)                                 ASQuarter,

                                DATENAME(MONTH,Date)                                  ASMonthName,

                                LEFT(DATENAME(MM,Date),3)                           AS MonthNameAbr,

                                DATEADD(DAY,-1,DATEADD(MONTH, 1,DATEADD(DAY,(DATEPART(DAY,

DATE)*-1)+1,Date)))                         AS MonthEndDate,

                                DATEADD(DAY,-1,DATEADD(MONTH, 1 + 12 -(DATEPART(MONTH,DATE)

- @FiscalStart + 1 +CASEWHEN @FiscalStart >DATEPART(MONTH,DATE)

THEN 12 ELSE 0 END),DATEADD(DAY,(DATEPART(DAY,DATE)*-1)+1,

Date)))                                                   AS YearEndDate,

                                DATEPART(MONTH,DATE)- @FiscalStart + 1 +

                                                CASEWHEN @FiscalStart >DATEPART(MONTH,DATE)THEN 12

ELSE 0 END                                             AS FiscalPeriod,

                                YEAR(DATEADD(DAY,-1,DATEADD(MONTH, 1 + 12 -(DATEPART(MONTH,

DATE)- @FiscalStart + 1 +CASEWHEN @FiscalStart >

DATEPART(MONTH,DATE)THEN 12 ELSE 0 END),DATEADD(DAY,

(DATEPART(DAY,DATE)*-1)+1,Date))))

AS FiscalYearEnd,

                                CASEDATEPART(MONTH,DATE)- @FiscalStart + 1 +CASEWHEN

@FiscalStart >DATEPART(MONTH,DATE)THEN 12 ELSE 0 END

                                                                WHEN 1 THEN

WHEN 2 THEN

WHEN 3 THEN 1

                                                                WHEN 4 THEN

WHEN 5 THEN

WHEN 6 THEN 2

                                                                WHEN 7 THEN

WHEN 8 THEN

WHEN 9 THEN 3

                                                                WHEN 10 THEN

WHEN 11 THEN

WHEN 12 THEN 4

                                                                END                                         AS FiscalQuarter

 

                FROM

                CTE_DatesTable

 

                OPTION (MAXRECURSION 10000)

Again, you can simply copy and paste the script from THIS ATTACHMENT into query editor.  However, you need to set a parameter to tell the script when your fiscal year starts.  Towards the top of the script, you’ll see a command “SET @FiscalStart = “ and you need to enter a month number from 1 to 12 where 1 is January through to 12 being December.  Therefore, if the first month of your financial year is April, set this value to 4.  Otherwise, the script works the same as the previous example but returns some additional Fiscal Time elements.

And it’s as simple as that!  Thanks Scott – truly brilliant!


SQL UNPIVOT Makes My Data Skinny

November 29, 2011

 

Guest post by David Churchward

If you’re anything like me, you will have eagerly sucked data into PowerPivot, keen to get cracking with solving all of your reporting needs.  If you’re an Accountant (amongst others) doing this, you will no doubt have brought your trial balance dataset in and found endless columns of values representing monthly values.  “How do I get time intelligence to work with this?” is probably one of the first issues you’ve encountered and then, further down the line, when you’ve conjured up complex measures to resolve this problem, the words “why can’t I get this to run quicker” pass your lips.  On this note, everyone should pay close attention to Less Columns, More Rows = More Speed! where Rob proves that the tall and skinny table is much more effective than a short and wide version.

In this post, I’ll explain how SQL can solve this problem by transforming the data on the data extraction process using a SQL function call UNPIVOT and it’s available to everyone – not just developers!

This is PowerPivot using Pivot tables – why on earth do I want to UNPIVOT.  Just like a kid who ties their shoelaces wrong, Mum has to untie in order to do it properly!  The fact is that data transformation has to happen in many situations to leverage the full power that PowerPivot can provide.

What’s this SQL Stuff – Surely that’s for Developers?

I was reading Rob’s recent posts on Why PowerPivot is Better Fed From a Database and it made me realise that there’s a grey area that represents how far an Excel Pro can go in data transformation.  I’m going to leave the “Developer V Excel Pro” debate to Rob (but believe me, he’s right).  As an Accountant who learnt SQL but who is an Excel Pro through and through, my view is that Excel Pros should be able to transform their data to the extent that a SQL view can.

Some of you will be aware, but others won’t, that there is a SQL query editor in PowerPivot.  If I’m upsetting Developers by revealing this, I’m sorry, but, if you’re worried, please take this as an opportunity to provide your Excel Pros with a secure Dev environment!

Accessing The SQL Query Editor

You can access query editor either when you first import data or by editing a table that you’ve already imported.

Accessing Query Editor on First Import

  1. From the PowerPivot window, select From Database > From SQL Serverand set up your connection as you would do normally.
  2. Click Next and select the option Write a query that will specify the data to import

image

Accessing Query Editor From a Table Already Imported

  1. Select the table that you want to affect in the PowerPivot Window and select Design > Table Properties
  2. Change from Table Preview to Query Editor

image

image

And there you have it.  A script that is running to import the data that you’ve specified.  We can now edit this to transform the data.

Before I go ahead, I just want to set a benchmark regarding the data that I’m using.  When I’ve brought this data in, I’ve got 38,183 records using 14 columns.  We’ll see what the UNPIVOT does to this in the end.  Everyone’s expecting a lot more rows but fewer columns, right?

image

The SQL “UNPIVOT” SCRIPT

So, here it is, the script to use in Query Editor mode:

SELECT

– These are the fields that I want in my final dataset

NCODE,

NNAME,

VALUE_CATEGORY, – NEW FIELD 1 – The field that takes my value column names

VALUE – NEW FIELD 2 – The field that takes my values from those columns

FROM

– I specify where these fields come from

(

SELECT

NCODE,

NNAME,

NTURNOVER_C1, – Value Field

NTURNOVER_C2, – Value Field

NTURNOVER_C3, — Value Field

NTURNOVER_C4, – Value Field

NTURNOVER_C5, – Value Field

NTURNOVER_C6, – Value Field

NTURNOVER_C7, – Value Field

NTURNOVER_C8, – Value Field

NTURNOVER_C9, – Value Field

NTURNOVER_C10, – Value Field

NTURNOVER_C11, — Value Field

NTURNOVER_C12 – Value Field

 

FROM dbo.AZZ_V_POWERPIVOTPRO_TB –Substitute this with the name of your source table or view

) PIVOT_TABLE

– Unpivot the table               

UNPIVOT

(VALUE FOR VALUE_CATEGORY in        – These are the names given to the

       — value field (NEW FIELD 1) and value column name (NEW FIELD 2)

(

– Enter value fields as per the select above

NTURNOVER_C1,

NTURNOVER_C2,

NTURNOVER_C3,

NTURNOVER_C4,

NTURNOVER_C5,

NTURNOVER_C6,

NTURNOVER_C7,

NTURNOVER_C8,

NTURNOVER_C9,

NTURNOVER_C10,

NTURNOVER_C11,

NTURNOVER_C12

)

) AS UNPVT

WHERE VALUE <> 0

ORDER BY NCODE, VALUE_CATEGORY

Note – green text within the script are comments to provide some direction as you use this script.  You can copy these into query editor too – PowerPivot will ignore them.

How to Use This Script

  1. Copy and paste the script into your query editor
  2. In the first SELECT, enter the field names that you require in your final dataset.  These should correlate to fields in the table except where I have noted “NEW FIELD 1” and “NEW FIELD 2”.  These “NEW FIELDS” will carry the value elements that we are unpivoting.
  3. In the second SELECT, detail the fields that you have already detailed in the first select together with the column names carrying the values that you will unpivot.  Also change the table or view name in the FROM clause to reference the source data that you are using.
  4. In the UNPIVOT section, detail all of the value fields again that you are unpivoting.
  5. In the ORDER BY clause, select the fields that you wish to use for dataset ordering.

When you save this, your dataset will have been narrowed and you’ll have a new field denoting the data type (or value column reference).

image

You’ll notice that we’re down from 38,183 records to 7,103.  This won’t always happen.  The reality is that we’re able to strip out the zero value cells by using the WHERE VALUE <> 0 clause.  If you’re carrying a number of cells that are zero valued, we won’t see that item as a record in this new dataset.  As a result, we get fewer records in this case.  In any event, your data is now a lot more streamlined and ready for more efficient measures to be written.  You don’t have to include the WHERE clause if you prefer, for whatever reason, to bring all records in.  If that’s the case, just delete that clause.

One Further Step

To make this truly useful, we need to translate VALUE_CATEGORY into something more meaningful.  I would normally expect this to be a date but most trial balances use column names such as “CURRENT1” so that the system isn’t date dependent.  Another table would be required that the VALUE_CATEGORY column can link through to which translates that value, thereby opening up the whole time intelligence functionality.

A Quick Warning

This UNPIVOT approach has given you the power to transform your data into a more streamlined dataset.  However, by using this approach you’ve created a local version of the script.  You won’t be able to reuse this without copying and pasting it into other workbooks (or at least I haven’t found a way).  However, those Developer friends of ours can give you a centralised version that can be controlled and reused more easily.  If you ask these guys nicely, they might create a view or stored proc to do this for you.  I’ve found with my team that salt based snacks, chocolate and the occasional Spag Bol for lunch does the trick and, if it doesn’t, I go ahead and do it anyway!


Why PowerPivot is Better Fed from a Database, Pt2

November 24, 2011

 
image

Think of a Database as a Bodybuilding Supplement
for Your PowerPivot Regimen

In part one I covered the following three benefits of pulling data into PowerPivot from a database, as opposed to using other sources such as Excel itself:

  1. Data-shaping – much easier in a database than in basically any other tool, this is the first benefit you will see.
  2. Auto-refresh – by removing manual data shaping from the system, you can then rely 100% on PowerPivot’s server-side automatic refresh to keep your reports up to date.  The first time in “Excel history” that Excel pros aren’t slaves to their own spreadsheets.
  3. More quality, less errors – the combination of PowerPivot’s “portable formulas” and performing your data shaping in a database greatly reduces the opportunity to make mistakes.

OK, time for benefits 4-7.

Benefit #4:  Complex Calc Columns

Let’s say you want to stamp your customers into quintiles – the top 20% in terms of dollars spent, the bottom 20%, etc.

And you don’t want that to be dynamically re-calculated every time you slice or filter a pivot – you just want it statically calculated according to total overall sales, so that you can then use the quintile as a slicer or row/column axis in your pivot.  You want to be able to offer a pivot report like this:

image

To do that, you need a calculated column in your Customers table:

image

But that’s a reasonably tricky calculated column, as it has to be calculated per-customer, against their matching rows in the Sales table, and then ranked against other all other customers.  If you do that in Excel, you’ve introduced another manual step which foils auto-refresh and in turn introduces more chances for error, etc. 

And doing it in PowerPivot is awkward, at least in v1.  I’m not saying you can’t do it in PowerPivot, because you can, and that DOES auto-refresh.  But it’s just so much easier in a database, if you have someone around who knows databases, and there are more complex examples than this one which get even harder in PowerPivot.

Benefit #5:  Centralized, Re-Useable Logic

This one is both its own distinct benefit as well as a multiplier for other benefits (such as data-shaping and complex calcs).  Anything you do in Excel is manual.  Anything you do in PowerPivot is “tied” to the workbook in which you did it.  But logic that you put in a database can be re-used across many different workbooks.

Which, of course, is very nice.  No need to re-write it every time.  No chance that it accidentally diverges from other models (a source of error).  And if you need to modify that logic in the future, you only have to do it in one place.

At this point it’s worth pointing out that a PowerPivot BI environment DOES share some characteristics with a traditional BI environment.  These db-centric benefits are a hallmark of traditional BI, and these are the things you want to preserve from the traditional approach.

Benefit #6:  “Magic” Sliding Windows and Parameters

This is one of my favorites.  Oftentimes, an organization operates according to a calendar, or even multiple calendars, that are not quite the same as the calendar hanging on your wall.  And that leads you to a place where your measures like “growth versus prior year” can get pretty complex.

I won’t go into great detail here because it would take awhile, but imagine a slicer driven from a table that is purely calculated in the database before input:

image

To get a sense of some similar techniques, which revolve around using “unconnected” slicer tables as inputs, see this post and this post.

Benefit #7:  Compression

It’s a fact:  imported columns compress much better in PowerPivot than calculated columns.  So if you have a choice between doing a calc column in your db (and then importing it) or importing a table and then adding that calc column via a DAX calc column in PowerPivot, DO IT IN THE DATABASE!

Better compression means smaller files, less memory consumption, and often faster response times to slicer clicks.  (Note that the bigger the table, the bigger the difference you will see).

Again, deserving of its own post in the future.


Delayed today, but post still coming

November 24, 2011

It’s Thanksgiving today in the US, and I’ve been taking it a bit easy this morning.  Plus my wife talked me into playing ice hockey with her last night and I did a lot of this last night:

image

Anyway, part two of “why PowerPivot is better fed from a database” is coming later today.  The blog is on an eleven-week streak of consistent Tuesday/Thursday posts and I’m not about to mess that up Smile


Why PowerPivot is Better Fed From a Database, Pt 1

November 22, 2011

An Excel Pro’s Two Year Journey in Database Land

In a post last week I mentioned that when you use a real database as a source for PowerPivot, a number of unexpected doors open up.  As a longtime Excel pro who has spent the past two years working closely with a database team, I can tell you that it’s been an eye opening experience.

So I have been advocating to Excel/PowerPivot pros for quite some time that they cozy up to their database teams (if they have them).  Of course, this works both ways, and I have also been advocating to Database/BI pros that they embrace and cooperate with Excel/PowerPivot pros.

Awareness, Flexibility, and Sometimes…  Outright Resistance

It’s an interesting ambassadorship.  On one hand, Excel pros are often accustomed to doing everything on their own, with no external help whatsoever, and so it’s mostly a matter of opening their eyes to benefits they’ve never imagined. 

But the other camp often doesn’t expect to learn anything new and relevant about the role of Excel, that unruly little brother of “real” BI.  In conversation, that camp splits into two subgroups rather quickly – one that is willing to consider new ideas and possibilities, and another that regards Excel as the devil.  Interactions with that second group can get ugly.  I also tend to fear for them a bit.

Today, I’m going to focus on the Excel crowd and explain why a db pro can make a huge difference in their lives.  While I have danced around the topic for two years on the blog, this will be my attempt to provide a definitive list of the benefits.

And you die-hard Excel haters out there, don’t worry, I’m coming back around to you soon Winking smile

Benefit #1:  Data Shaping is Easier in Databases.  MUCH Easier.

Hey, Excel is just a collection of individual cells when you come right down to it.  Cells go into formulas, and cells come out.  But turning a wide and short table into a tall skinny table (or vice versa) for instance can consume the better part of an Excel pro’s day in some cases.  That same operation might take a db pro 5 minutes or less.  And even better, the next time might take them no time at all…

Benefit #2:  Auto-Refresh!

Once the db pro has a script in place (or query or view or sproc or whatever it is that they deem best), it can now be run automatically in response to your refresh request from PowerPivot. 

If you are running PowerPivot on your desktop, hey, now it’s just one click (refresh) to pull in the latest, properly-shaped data.  But even better, if you have access to a PowerPivot-enabled SharePoint server, you can put your workbook up there and schedule it to refresh itself!

Even if the manual shaping that you do today only takes you a few minutes each time, it’s still worth it to outsource it to a db.  The SharePoint option means you don’t have to do anything each day to merely update the reports, and the difference between “small” and “nothing” is huge.  Like, “you can go on vacation without repercussions” huge.  Or “you don’t have to come in early each day” huge.

Benefit #3:  Quality

Did you know there are entire conferences devoted to the topic of spreadsheet errors?  Spreadsheet errors are a fact of life in traditional spreadsheets, but they are primarily due to specific problems that a good PowerPivot “ecosystem” eliminates.  One is the lack of convenient named reference – “what did D$14 refer to again?” crosses the inner monologues of Excel pros worldwide millions of times a day.  PowerPivot fixes that – everything is referenced by table/column/measure name.

But raw repetition is the real killer.  If you perform the same spreadsheet task every day for a year, pure statistics tells us you will make mistakes.  And if the task is tedious, you will make even more.

The lack of “portable formulas” in traditional spreadsheets is an underappreciated source of repetition.  Even a single iteration of a reporting task explodes into repetitious subtasks and provides lots of opportunity for error.  (PowerPivot fixes that one too).

So we are left with raw repetition – performing the same task every day – as our primary source of error.  Once you have PowerPivot, data shaping is the biggest source of repetition.  Outsourcing that shaping logic into a database, then, doesn’t just save time.  It prevents mistakes. 

Why?  Because the db logic is written only once, and it doesn’t change no matter how many times you run it.  Even if there were mistakes made in the db logic, you will catch them sooner or later (usually immediately), and once you fix them, they stay fixed.

Come back Thursday for part two, with items four through seven.


Comparing Access to PowerPivot

November 17, 2011

 
I don't think the artist had data apps in mind, but this pic rocks!

Here’s a question that comes up with increasing frequency:  “PowerPivot seems kinda similar to Access in many ways, what’s the difference?”

Why Does the Question Come Up?

Some of you are thinking “yeah, I can see why people would ask that.”  And others of you are thinking “WHAT???  That questions makes ZERO sense, they are NOTHING alike!”

You are both right.

The biggest reason why the question comes up, I think, is the longstanding symbiosis between Access and Excel – Access as data source, Excel as analysis tool.  Remember, Excel used to be limited to about 64,000 rows of data.  And many Excel pros learned to import large data sets into Access rather than Excel, manipulate and prep the data in Access, and THEN import from Access into Excel for PivotTables, charting, etc.

With Excel expanding from 64k to 1M rows in 2007, that tradition has already begun to fade, to an extent.  But even for data sets that fit into Excel, there is still a good reason that drives Excel pros into Access:  VLOOKUP is slow.

VLOOKUP = Excel Acting Like a Database, and Driving Folks to Access

Even for large data sets, arithmetic calculations in Excel can be blindingly fast.  After all, Excel is designed for that.  But VLOOKUP, and its more advanced cousin INDEX/MATCH, is not arithmetic.  It’s a search – “go find me a value that looks like X, and when you find it, return value Y from the same row.”

Even when you’re dealing with row counts merely in the thousands, that can get slow in Excel.  Because “search and retrieve” is what databases are designed for.  And Excel is not a database.  When it comes to finding values, Excel isn’t terribly more efficient than Word.  (A risky thing for me to say, I expect to be corrected in three…  two…  )

But since many data sets inherently “arrive” as multiple separate tables, you can’t avoid trying to splice them together, and that means VLOOKUP, or using a real database product.  I know the SQL snobs will say that Access doesn’t qualify, but Access IS a real db.

An Understandable Question

With that in mind, it’s easy to see why longtime Excel pros see their first PowerPivot demo, and come away asking this question.  At a high level, this is what they see:

Access Versus PowerPivot

OK, so what’s the answer?

PowerPivot Does Things That Access Will Never Do

You had me at DAX MeasuresThe first time you write a “Sort By Slicers” or an “Iffer-Blanker” or  a Set of Greater/Less Than Slicers or a Custom Calendar Running Total or even just worked with ALL(), you know this isn’t Access.  Oh, and you are doing that in Excel pivots, not in a separate window.

There are other benefits of course, for sure, but DAX Measures are a gamechanger.  Let’s leave it at that for now.

And Yes, You Can Replace “Access as Data Source” With PowerPivot*

The two biggest reasons that drove Excel pros into Access in the past are in fact alleviated with PowerPivot.  No 64K row limit in PowerPivot.  No 1M row limit either.  I commonly demo a 300M PowerPivot row workbook on my laptop!  You can load a lot more data into PowerPivot than you can into Access.

And VLOOKUP isn’t something you even need anymore in PowerPivot.  Got multiple tables?  Fine!  Leave them as separate tables, link them via relationships, and you are done.  That’s not even just a convenience – leaving them as separate tables is actually even just better, for many reasons.

All of that “go find me a match in another table” stuff is taken care of by PowerPivot.  Lightning fast in fact.

But whoa there, I put an asterisk on that statement above.  For good reason.

PowerPivot Goes Better with Databases!

image

Database –> PowerPivot –> Excel = Happy Happy

Fact is, if you work in an organization that uses SQL Server or another industrial strength database product, you are better off connecting PowerPivot to that db.

Or more accurately, you will get even more out of PowerPivot if you have the cooperation of a database professional.

Why is that?  I can (and will) write many posts on that.  For now let’s keep it simple and just point out two reasons:  1) Databases are inherently a very good place to do data “shaping,” which is not something you can do at all in PowerPivot.   and 2) Databases are great places to perform complex row-wise and cross-row business calcs.  They centralize those calcs for re-use, often take the db pro 5 mins to do versus much longer for you, and result in faster and more compact workbooks than if you use calc columns.

I’m not saying you need a db pro to get amazing things out of PowerPivot.  But there’s another level even beyond amazing, and it opens up when you cooperate with a db pro.

Final Note:  Be Thankful PowerPivot Wasn’t Built By Office

I say this because it would have been questioned to death.  It’s hard to imagine, but as a product like PowerPivot is taking shape at Microsoft, no one is really sure how to describe it yet, or even what it’s going to turn out to truly be.  A hundred people in the Office org would have had the same question – are we cannibalizing the Access business, and there would have been as much time spent answering that as designing the actual product.

Now, in hindsight, no one in Office is worried about that.  Access always had a much bigger mission than carrying around data for Excel.  And Access’s current mission has evolved quite a bit from what it was even a few years ago.

But those inevitable nagging questions early on would have saddled PowerPivot with a number of “thou shalt not cross this line” concessions.  Concessions which ultimately were not needed, and that would have hurt the product.

Gives new meaning to the term “Office Politics.”