Using Query Editor to Create a Time Table

December 1, 2011

 

Guest Post by David Churchward with Technical Input from Scott Senkeresty

The first thing I ever do when I create a new PowerPivot workbook is create the date tables that I need.  Until now, I’ve used a preconditioned SQL table to draw this information from.  Now, thanks to Scott Senkeresty, you can now create your own – from PowerPivot!

Scott is a true genius.  He spends a lot of time under the hood of Pivotstream operations working his magic and wizardry and, every now and then, he pops up for air to deliver a masterpiece like this.

 

It’s as Simple as Copy and Paste

If you read my last post SQL UNPIVOT Makes My Data Skinny, you’ll know where query editor is and the sort of tricks that it opens up within PowerPivot.  In this post, there’s some SQL scripts that create time tables and you can simply cut and paste them into the query editor and use them straight away.

Having said this, it’s worth checking out Rob’s post The Ultimate Date Table to see why a date table is important and where you can get one of your own from Azure DataMarket.  However, I think we’ve gone one step better now and given you a script that you can parameterise easily, condition relatively easily and setup extremely easily.  Add to that the fact that it runs in mere seconds!

AND there’s one more bonus, especially for Accountants, you can set this up for YOUR Fiscal Year!

Copy, Paste, Save – Simples!

To get going, simply copy and paste the script from THIS ATTACHMENT into query editor.  When you save it – bam – time table complete.

DECLARE @FirstDate DATE

DECLARE @LastDate DATE

SET @FirstDate =’1 Jan 2008′– Enter first date of calendar as ‘dd mmm yyyy’

SET @LastDate =’31 Dec 2015′– Enter last date of calendar as ‘dd mmm yyyy’

 

IF @FirstDate ISNULLOR @FirstDate =

                BEGIN

                                SET @FirstDate =’1 Jan 2000′

                END

               

IF @LastDate ISNULLOR @LastDate =

                OR @LastDate < @FirstDate

                ORDATEDIFF(DAY, @FirstDate, @LastDate)> 10000

                BEGIN

                                SET @LastDate =DATEADD(DAY, 10000, @FirstDate)

                END

 

                ;WITH CTE_DatesTable

                AS

                (

                  SELECT @FirstDate AS [Date]

         UNIONALL

         SELECTDATEADD(DAY, 1, [Date])

         FROM CTE_DatesTable

         WHEREDATEADD(DAY, 1, [Date])<= @LastDate

                )

 

                SELECT

                                Date,

                                DATEPART(DAYOFYEAR,Date)                             ASDayOfYear,

                                DATEPART(DAY,Date)                                          ASDay,

                                DATEPART(MONTH,Date)                                    ASMonth,

                                DATEPART(YEAR,Date)                                        ASYear,

                                DATENAME(WEEKDAY,Date)                               ASDayOfWeek,

                                LEFT(DATENAME(WEEKDAY,Date),3)                 AS DayOfWeekAbr,

                                DATEPART(WEEK,Date)                                       AS WeekOfYear,

                                DATEPART(QUARTER,Date)                                 ASQuarter,

                                DATENAME(MONTH,Date)                                  ASMonthName,

                                LEFT(DATENAME(MM,Date),3)                           AS MonthNameAbr,

                                DATEADD(DAY,-1,DATEADD(MONTH, 1,DATEADD(DAY,(DATEPART(DAY,

DATE)*-1)+1,Date)))                         AS MonthEndDate

 

                FROM

                CTE_DatesTable

 

                OPTION (MAXRECURSION 10000)

Without doing anything to this script, you’ll get a time table that spans Jan 2008 to Dec 2015.  That could be enough.  In which case, you’re done.  However, I’m sure you want more.

Change Date Parameters to Widen the Timeframe

The great news with this script is that it’s so easy to adjust to cover the timeframe that you need.  You just need to follow these simple couple of steps:

  1. Towards the top of the script you’ll see “SET @FirstDate = “.  You simply need to change the value to the first date in your dataset (or perhaps an earlier date to be sure).
  2. I’m sure you’ve already guessed step 2 – change “SET @LastDate = “ to a value that is beyond the last date in your dataset.

I almost wish I could make it more difficult but Scott’s nailed this so well that it’s almost impossible to get wrong – well almost!

A Quick Health Warning

This script uses a recursive CTE (Common Table Expression).  If you’re not careful, it’s very possible to spin this baby into an infinite cycle.  Without going into why this happens, here’s a couple of points to make sure that you don’t fall into that trap:

  1. The IF – BEGIN – SET – END statements are there to ensure that valid entries are made for @FirstDate and @LastDate.  It’s probably best to leave these in.
  2. At the end of the script, there’s a command OPTION(MAXRECURSION 10000).  This ensures that the script won’t loop more than 10,000 times (ie 10,000 days).  If you set the recursion to zero, this removes all limits and then you could get into an infinite cycle.

Don’t be scared to give it a go.  There’s controls here to stop anything going wrong and what’s the worst that could happen anyway!

The Next Level – Fiscal Elements

In all my time using SQL and PowerPivot, I’m yet to work in an environment where a year runs from 1st January to 31st December.  As a result, we refer to a Fiscal Year to represent this alternate calendar.  I took the liberty of hacking Scott’s script to add these elements in.

DECLARE @FirstDate DATE

DECLARE @LastDate DATE

DECLARE @FiscalStart INT

SET @FirstDate =’1 Jan 2008′– Enter first date of calendar as ‘dd mmm yyyy’

SET @LastDate =’31 Dec 2015′– Enter last date of calendar as ‘dd mmm yyyy’

SET @FiscalStart = 4 – First calendar period of Fical Year – 1=Jan, 2=Feb…., 12=Dec

 

IF @FirstDate ISNULLOR @FirstDate =

                BEGIN

                                SET @FirstDate =’1 Jan 2000′

                END

               

IF @LastDate ISNULLOR @LastDate =

                OR @LastDate < @FirstDate

                ORDATEDIFF(DAY, @FirstDate, @LastDate)> 10000

                BEGIN

                                SET @LastDate =DATEADD(DAY, 10000, @FirstDate)

                END

 

                ;WITH CTE_DatesTable

                AS

                (

                  SELECT @FirstDate AS [Date]

                 UNIONALL

                 SELECTDATEADD(DAY, 1, [Date])

                 FROM CTE_DatesTable

                 WHEREDATEADD(DAY, 1, [Date])<= @LastDate

                )

 

                SELECT

                                Date,

                                DATEPART(DAYOFYEAR,Date)                             ASDayOfYear,

                                DATEPART(DAY,Date)                                          ASDay,

                                DATEPART(MONTH,Date)                                    ASMonth,

                                DATEPART(YEAR,Date)                                        ASYear,

                                DATENAME(WEEKDAY,Date)                               ASDayOfWeek,

                                LEFT(DATENAME(WEEKDAY,Date),3)                 AS DayOfWeekAbr,

                                DATEPART(WEEK,Date)                                       AS WeekOfYear,

                                DATEPART(QUARTER,Date)                                 ASQuarter,

                                DATENAME(MONTH,Date)                                  ASMonthName,

                                LEFT(DATENAME(MM,Date),3)                           AS MonthNameAbr,

                                DATEADD(DAY,-1,DATEADD(MONTH, 1,DATEADD(DAY,(DATEPART(DAY,

DATE)*-1)+1,Date)))                         AS MonthEndDate,

                                DATEADD(DAY,-1,DATEADD(MONTH, 1 + 12 -(DATEPART(MONTH,DATE)

- @FiscalStart + 1 +CASEWHEN @FiscalStart >DATEPART(MONTH,DATE)

THEN 12 ELSE 0 END),DATEADD(DAY,(DATEPART(DAY,DATE)*-1)+1,

Date)))                                                   AS YearEndDate,

                                DATEPART(MONTH,DATE)- @FiscalStart + 1 +

                                                CASEWHEN @FiscalStart >DATEPART(MONTH,DATE)THEN 12

ELSE 0 END                                             AS FiscalPeriod,

                                YEAR(DATEADD(DAY,-1,DATEADD(MONTH, 1 + 12 -(DATEPART(MONTH,

DATE)- @FiscalStart + 1 +CASEWHEN @FiscalStart >

DATEPART(MONTH,DATE)THEN 12 ELSE 0 END),DATEADD(DAY,

(DATEPART(DAY,DATE)*-1)+1,Date))))

AS FiscalYearEnd,

                                CASEDATEPART(MONTH,DATE)- @FiscalStart + 1 +CASEWHEN

@FiscalStart >DATEPART(MONTH,DATE)THEN 12 ELSE 0 END

                                                                WHEN 1 THEN

WHEN 2 THEN

WHEN 3 THEN 1

                                                                WHEN 4 THEN

WHEN 5 THEN

WHEN 6 THEN 2

                                                                WHEN 7 THEN

WHEN 8 THEN

WHEN 9 THEN 3

                                                                WHEN 10 THEN

WHEN 11 THEN

WHEN 12 THEN 4

                                                                END                                         AS FiscalQuarter

 

                FROM

                CTE_DatesTable

 

                OPTION (MAXRECURSION 10000)

Again, you can simply copy and paste the script from THIS ATTACHMENT into query editor.  However, you need to set a parameter to tell the script when your fiscal year starts.  Towards the top of the script, you’ll see a command “SET @FiscalStart = “ and you need to enter a month number from 1 to 12 where 1 is January through to 12 being December.  Therefore, if the first month of your financial year is April, set this value to 4.  Otherwise, the script works the same as the previous example but returns some additional Fiscal Time elements.

And it’s as simple as that!  Thanks Scott – truly brilliant!


SQL UNPIVOT Makes My Data Skinny

November 29, 2011

 

Guest post by David Churchward

If you’re anything like me, you will have eagerly sucked data into PowerPivot, keen to get cracking with solving all of your reporting needs.  If you’re an Accountant (amongst others) doing this, you will no doubt have brought your trial balance dataset in and found endless columns of values representing monthly values.  “How do I get time intelligence to work with this?” is probably one of the first issues you’ve encountered and then, further down the line, when you’ve conjured up complex measures to resolve this problem, the words “why can’t I get this to run quicker” pass your lips.  On this note, everyone should pay close attention to Less Columns, More Rows = More Speed! where Rob proves that the tall and skinny table is much more effective than a short and wide version.

In this post, I’ll explain how SQL can solve this problem by transforming the data on the data extraction process using a SQL function call UNPIVOT and it’s available to everyone – not just developers!

This is PowerPivot using Pivot tables – why on earth do I want to UNPIVOT.  Just like a kid who ties their shoelaces wrong, Mum has to untie in order to do it properly!  The fact is that data transformation has to happen in many situations to leverage the full power that PowerPivot can provide.

What’s this SQL Stuff – Surely that’s for Developers?

I was reading Rob’s recent posts on Why PowerPivot is Better Fed From a Database and it made me realise that there’s a grey area that represents how far an Excel Pro can go in data transformation.  I’m going to leave the “Developer V Excel Pro” debate to Rob (but believe me, he’s right).  As an Accountant who learnt SQL but who is an Excel Pro through and through, my view is that Excel Pros should be able to transform their data to the extent that a SQL view can.

Some of you will be aware, but others won’t, that there is a SQL query editor in PowerPivot.  If I’m upsetting Developers by revealing this, I’m sorry, but, if you’re worried, please take this as an opportunity to provide your Excel Pros with a secure Dev environment!

Accessing The SQL Query Editor

You can access query editor either when you first import data or by editing a table that you’ve already imported.

Accessing Query Editor on First Import

  1. From the PowerPivot window, select From Database > From SQL Serverand set up your connection as you would do normally.
  2. Click Next and select the option Write a query that will specify the data to import

image

Accessing Query Editor From a Table Already Imported

  1. Select the table that you want to affect in the PowerPivot Window and select Design > Table Properties
  2. Change from Table Preview to Query Editor

image

image

And there you have it.  A script that is running to import the data that you’ve specified.  We can now edit this to transform the data.

Before I go ahead, I just want to set a benchmark regarding the data that I’m using.  When I’ve brought this data in, I’ve got 38,183 records using 14 columns.  We’ll see what the UNPIVOT does to this in the end.  Everyone’s expecting a lot more rows but fewer columns, right?

image

The SQL “UNPIVOT” SCRIPT

So, here it is, the script to use in Query Editor mode:

SELECT

– These are the fields that I want in my final dataset

NCODE,

NNAME,

VALUE_CATEGORY, – NEW FIELD 1 – The field that takes my value column names

VALUE – NEW FIELD 2 – The field that takes my values from those columns

FROM

– I specify where these fields come from

(

SELECT

NCODE,

NNAME,

NTURNOVER_C1, – Value Field

NTURNOVER_C2, – Value Field

NTURNOVER_C3, — Value Field

NTURNOVER_C4, – Value Field

NTURNOVER_C5, – Value Field

NTURNOVER_C6, – Value Field

NTURNOVER_C7, – Value Field

NTURNOVER_C8, – Value Field

NTURNOVER_C9, – Value Field

NTURNOVER_C10, – Value Field

NTURNOVER_C11, — Value Field

NTURNOVER_C12 – Value Field

 

FROM dbo.AZZ_V_POWERPIVOTPRO_TB –Substitute this with the name of your source table or view

) PIVOT_TABLE

– Unpivot the table               

UNPIVOT

(VALUE FOR VALUE_CATEGORY in        – These are the names given to the

       — value field (NEW FIELD 1) and value column name (NEW FIELD 2)

(

– Enter value fields as per the select above

NTURNOVER_C1,

NTURNOVER_C2,

NTURNOVER_C3,

NTURNOVER_C4,

NTURNOVER_C5,

NTURNOVER_C6,

NTURNOVER_C7,

NTURNOVER_C8,

NTURNOVER_C9,

NTURNOVER_C10,

NTURNOVER_C11,

NTURNOVER_C12

)

) AS UNPVT

WHERE VALUE <> 0

ORDER BY NCODE, VALUE_CATEGORY

Note – green text within the script are comments to provide some direction as you use this script.  You can copy these into query editor too – PowerPivot will ignore them.

How to Use This Script

  1. Copy and paste the script into your query editor
  2. In the first SELECT, enter the field names that you require in your final dataset.  These should correlate to fields in the table except where I have noted “NEW FIELD 1” and “NEW FIELD 2”.  These “NEW FIELDS” will carry the value elements that we are unpivoting.
  3. In the second SELECT, detail the fields that you have already detailed in the first select together with the column names carrying the values that you will unpivot.  Also change the table or view name in the FROM clause to reference the source data that you are using.
  4. In the UNPIVOT section, detail all of the value fields again that you are unpivoting.
  5. In the ORDER BY clause, select the fields that you wish to use for dataset ordering.

When you save this, your dataset will have been narrowed and you’ll have a new field denoting the data type (or value column reference).

image

You’ll notice that we’re down from 38,183 records to 7,103.  This won’t always happen.  The reality is that we’re able to strip out the zero value cells by using the WHERE VALUE <> 0 clause.  If you’re carrying a number of cells that are zero valued, we won’t see that item as a record in this new dataset.  As a result, we get fewer records in this case.  In any event, your data is now a lot more streamlined and ready for more efficient measures to be written.  You don’t have to include the WHERE clause if you prefer, for whatever reason, to bring all records in.  If that’s the case, just delete that clause.

One Further Step

To make this truly useful, we need to translate VALUE_CATEGORY into something more meaningful.  I would normally expect this to be a date but most trial balances use column names such as “CURRENT1” so that the system isn’t date dependent.  Another table would be required that the VALUE_CATEGORY column can link through to which translates that value, thereby opening up the whole time intelligence functionality.

A Quick Warning

This UNPIVOT approach has given you the power to transform your data into a more streamlined dataset.  However, by using this approach you’ve created a local version of the script.  You won’t be able to reuse this without copying and pasting it into other workbooks (or at least I haven’t found a way).  However, those Developer friends of ours can give you a centralised version that can be controlled and reused more easily.  If you ask these guys nicely, they might create a view or stored proc to do this for you.  I’ve found with my team that salt based snacks, chocolate and the occasional Spag Bol for lunch does the trick and, if it doesn’t, I go ahead and do it anyway!