Comments on: Combine Multiple Worksheets/Workbooks into a Single PowerPivot Table

By: SAP user

SAP user — Thu, 04 Apr 2013 11:11:51 +0000

But if your office version is 32-bit it is apparently not possible to install the 64-bit add-in? Our IT department recently installed the powerpivot add-in however it does not work correctly, e.g. i cannot get data from different sources into powerpivot, only by copy and paste.
How do I find out if they installed the correct 32-bit add-in or is there a way to test it?

By: SAP user

SAP user — Thu, 04 Apr 2013 11:03:11 +0000

the example works fine but with my sheets I did not succeed. Although the sheets have identical headers however different number of rows connection was not possible. Is there a limit to the number of columns?

By: Alex

Alex — Wed, 20 Feb 2013 11:16:54 +0000

Thank you for your comment. I followed your suggestion:
1- Decrease number of columns from 35 to 28. Result- same file is not 4,2 GB it is now 3,37 GB.
2- Load .csv as text to PowerPivot- no problem. Now even faster.
Result: that particular problem- solved.
3- Increase .csv with 28 columns by *2 lines. Result- 28 columns X 30M lines, .csv=6,6GB
4- Load 6,6 GB.csv as text to PowerPivot. Same error as before. “an error occurred while loading the file” -> table import wizard -> unhandled exception occurred in a component in your application… object reference not set to an instance of an object”
I think it is obvious that it is impossible (in my case) to load .csv>4,00 GB into PowerPivot.
Why did I start using .csv solution?
Previous exercise with PowerPivot: -> load data via sql directly from source (ORACLE). Very heavy sql query with many tables. Case: query should be completed between 05.00 and 23.00 (tables updated between 23.00 and 05.00). in given time I can load max 4M lines, so combining multiple workbooks seems like a good alternative. Run sql in excel-> create data workbooks with <1M lines:
Option1 – “UNION ALL” SQL http://www.contextures.com/PowerPivot-Identical-Excel-Files.html for same reasons as above can’t combine more than 4 GB of excel books (in this case max 6 books or 5M lines X 36 columns;
Option2- use Rob’s example. Can combine more compressed .csv files (15M lines X 36 columns or app 20-22M lines X 28 columns.
Conclusion: (in my case) no matter how I doing it in 64-bit excel I am hitting the wall after 4 GB data source.
Question: how to load more lines with 28 columns within max 4GB data file? So fare .csv is the best, but kind of not good enough for my requirements.

By: powerpivotpro

powerpivotpro — Fri, 15 Feb 2013 14:52:16 +0000

36 columns is a lot. Do you need all of those? Can you import just a subset of them? PowerPivot has no problem with massive row counts but columns are a killer with large data sets. Also, if you have calculated columns added to that table (in PowerPivot), try deleting them, then importing the data, then adding the columns back. That sometimes works.

By: Alex

Alex — Fri, 15 Feb 2013 09:12:55 +0000

Hi, thank you for your reply. it fails when I tried to import CSV into PowerPivot. This particular CSV can hold 4,02 GB (app 16,5M lines X 36 columns)- no problem. I have 64-bit excel.
I am not a pro and start exercising PowerPivot only few months ago, but fail sharply after 4 GB looks really strange and therefore I refer to RAM-issue…

By: powerpivotpro

powerpivotpro — Thu, 14 Feb 2013 12:37:42 +0000

When does it fail? On import into PowerPivot, or on creation of the combined CSV?

Also, are you running 32 or 64-bit Excel? If 32 bit, Excel can only use something like 2 GB of RAM, no matter how much RAM is on your computer. I prefer 64-bit Excel purely because of PowerPivot.

By: Alex

Alex — Thu, 14 Feb 2013 11:56:59 +0000

Rob, this is superb. It is very nice and simple way of doing it. I exercised it with .csv files 2; 3 and up to 4 GB. Now, I hit 4,2 GB and it is not working. I have 16 GB RAM, so I can assume that local memory is not an issue… Any ideas why if .csv > 4GB it is impossible to import CSV into PowerPivot? Thank you.

By: Paddy

Paddy — Mon, 11 Feb 2013 12:09:54 +0000

Is there any way of doing this for Non-excel databases? I use Dynamics NAV & want to merge the Sales Line, Sales Invoice Line & Sales Cr.Memo line tables before creating a report from the combined table.

Many thanks in advance

By: David

David — Fri, 12 Oct 2012 18:06:35 +0000

This is pretty great. However, the remove header and/or saving each csv filter is still a manual process for each sheet. Group all sheets and deleting the row work might work for the remove header.

Any way to quickly save all sheets into separate csv files without a macro?

By: dan

dan — Wed, 26 Sep 2012 11:05:52 +0000

If you are more comfortable working within powerpivot and know a little SQL then the following post is quite useful : http://www.contextures.com/PowerPivot-Identical-Excel-Files.html
plus you don’t have to ‘save as’ with each file and use a cmd window.