Tuesday, April 15, 2014

Large Lists, BCS, Excel REST, JSOM, CSVs and Office 365 OH MY!

An interesting challenge came across my desk last week:
"How can we import a CSV into Office 365 and use that data to tag other items within the site.  The CSV currently has ~7600 rows and is expected to grow.  We'll also need to re-import the CSV on an ad-hoc basis when the data changes."
The last part of that was the real issue.  There was simply no easy way of doing that.  After trying a few things, falling flat on my face, I'm prepared to share my exploration into the different options.

Each have their own merits and pitfalls.  This post will examine each and try to shed some light on the pitfalls I've found using each of these.


Wiring up an ECT on Office 365 can be a little finicky. I initially had some issues due to 2 BDC Models that were created for the same ECT.  After calling in the eagle-eyed BCS guru, +Fabian Williams, I got squared away.

Immediately after that, I could tell that I was not going to be able to use BCS for what I needed.  In Office 365, there is a hard limit of 2000 items that can be retrieved.  Ironically, the error message that is displayed is not a supported cmdlet for Office 365.

Office 365 BCS Throttle Error
Adding insult to injury, I decided to run a simple test using JSOM.  I wanted to clarify if JSOM would provide me PagingInfo with a BCS List.  Using the code below, you'd expect line 68 to produce a value instead of nothing at all.

Since BCS will not work and due to the limitation of the API, I had to look for another solution.

Custom List

Using a custom list initially worked great.  I'm able to use JSOM, query the list for 5000 items per trip to the server, AND get PagingInfo.  Using the code below works great for this scenario.

Importing the Excel into Office 365 is relatively straightforward and will work for most needs.  The file I used had about ~7600 rows of data.  After importing the file, I noticed the Server Resource Quota was tapped.

Office 365 Server Utilization
So using this approach has 2 problems.  I will not be able to do a mass import again of my data (the list already exists) and the Server Resource Quota points are exhausted.

Excel REST

This seemed like a cool way of getting around the limitations above, so I dove in to find out if this will work for my needs.  After all, I'm allowed to have a *lot* of rows in Excel and I'll be able to easily update the file, since it's in a document library.  Using the code below, I ran into a showstopper though.

There is a hard limit in the API set at 500 rows.  That would be painfully slow to get all of the items or even worse; a user may try to use the form control while this is still querying for data.

Excel REST API - 500 row limit
So that leaves us with our raw data that was exported from SQL and given to us to use.


Updating the CSV will be easy, since it will be stored in a document library.  Now all we need to be able to do is make sense of it.  Using the code below, I'm able to parse the CSV and create an array of objects that I need to pass off to another library.  Also note the use of localStorage.  This is a nice way to cache the data and prevent the retrieval/processing of the data client-side on every page load.  If the CSV is updated, simply clear the browser cache and you'll get the latest and greatest.


All approaches have their merits and pitfalls... BCS and PagingInfo, I'm looking at you!  If the ad-hoc mass-import wasn't needed, then using list driven data would have been my choice.  If I used that approach, I would have still used localStorage though.  It makes sense to cache the processed data since it'll not change very much.  Since my solution works client-side, I'll have to take into consideration the amount of time this takes to render.  I'm getting good performance out of the CSV approach, so I'm going to stick with it for the time being.

Friday, January 10, 2014

Fire Workflows with Initiation Parameters using #SPServices

Firing Workflows using Javascript, I've never had to pass in Initiation Parameters.  This post takes a look at how to do just that and provides some code that will allow easy use of Workflows in Javascript.

Setup Workflow

The workflow has to be set for Manual starting, otherwise this will not work.  Also, to pass in parameters to the workflow, you'll need to have Initiation Variable(s) within the workflow. I've only fiddled with Number and Single Line of Text fields, so if you use other column types, feel free to share your experience.

Workflow Initiation Parameters

This workflow is simply logging the variables out to the Workflow History.  Easy peazy...

Workflow Parameters

Looking over the documentation for SPServices and StartWorkflow, I found some examples that were a great starting point.  After fiddling a bit with 1 field, I decided to test this a little more.  I created a column with spaces in the Name field *gasp*.  I only did this to see how to handle this programmatically, so a word to the wise: Friends don't let friends create columns with spaces...

Reading over the examples, if you have multiple parameters, it says you have to change from passing the column name to this weird pattern: 

<Data><Parameter1>" + parameter1 + "</Parameter1><Parameter2>" + parameter2 + "</Parameter2></Data>

I've found this to not work at all for me at all [sad_panda]... Back to the drawing board, I guess. Then an idea came to me.  Since I'm targeting a column with a space in it, I tried what normally happens to spaces in Static Names: _x0020_.  So, I tried this next:

<Data><TextField>Will it blend?</TextField><With_x0020_Spaces>42</With_x0020_Spaces></Data>

However, this didn't work either!  Very curious to find a resolution, I set out to find out why this didn't work...  Using SPD (SharePoint Designer), you are able to view the files generated by the workflow. Opening up the XML file as text, you can clearly see that SPD removed the space in the Static Name.

Workflow wfconfig.xml
Within this file, all of the Initiation Parameters are visible and it's now easy to tell what's exactly going on.

Workflow Parameter Names
For all of this to work while using multiple parameters, you have to use the exact Static Name as defined in the XML.  The workflow parameters below work just fine for me now.

<Data><TextField>Will it blend?</TextField><WithSpaces>42</WithSpaces></Data>

Code to Fire the Workflow

This function will handle the pain of getting a workflow to fire.  All you need to know is the correct URL, the workflow name, and the workflow parameters ( if any ).

*** Update *** I took my original idea and made it more or less a plug-in for SPServices. Add this function to the SPServices source and it'll work without any issues. Original function: