Thursday, March 22, 2012

Dynamic packages and maxconcurrentexecutables

I have recently prototyped a system, in which I use meta data to build a package of execute package tasks.

Essentially, in a Script task I do the following:

1) Create a new package in memory.

2) I add variables, logging to this package.

3) To allow for precedences I create a sequence container for any and all execute pacakge tasks that can run in parallel.

4) To help with parameters specific to the execute package task, I add another sequnce container for the individual excute package task.

5) Finally I add the actual execute package task.

6) Save the package to local disk.

7) Execute this pacakge.

I hadn't been setting explicitly setting the MaxConcurrentExecutables, but upon opening the saved copy of the last package I built and executed it has the default -1 setting.

My problem is that while this in-memory package is running it appears to be only running a single executalbe at a time. I'm going to try setting maxconcurrentexecutable to 4 or some other number to see if I get some parallel execution going on.

The real question is "Is there a limitation on using dynamic packages that limits them to only run a single executable at a time?". I haven't found anything in BOL that leads me to believe there is, but It was very obvious that only one executable would run at a time when I test this out.

How are you testing this? If you run the dynamic generated package directly, do multiple executables run?|||Right now my testing involves watching the staging tables that I have. I expect to see them start populating with data somewhat simultaneously.|||Try opening the generated package, and running it directly through BIDS. There are a number of reasons why a package might not run concurrent executables (system resources, etc), but if it works in BIDS, then we can narrow it down to a code issue.

No comments:

Post a Comment