Dynamic OLE DB Source and Destination

Hi,

I am building SSIS for 3 different files that have identical
schema and mapping logic.

In my OLE DB Source (object name - "OLEDBSource_SourceTable")
Data Access mode is "Variable name".
As soon as I swithced to this Data Acces mode
it started to give me an error:

[OLEDBSource_SourceTable [1]] Warning: The external metadata column collection is out of synchronization
with the data source columns.

The column "DEAL_NUM" needs to be updated in the external metadata column collection.
The "external metadata column "DEAL_NUM_Flag" (34529)" needs to be removed
from the external metadata column collection.
The "external metadata column "recordID" (33740)"
needs to be removed from the external metadata column collection.
Meta data CANNOT change. You cannot setup an OLE DB source and have columns mapped and then just change the underlying source table. How can you imagine that working?

Search this forum for dynamic source/destinations and you'll get plenty of conversations on this topic.

Bottom line, SSIS relies on meta data. If you change it, how do you expect it to operate?|||

Hi Riga,

To make the data source dynamic you need to make the ConnectionString dynamic.

Hope this helps,

Andy

|||Everything turned out to be much easier, gentlemen!
As soon as I set in my OLE DB Source Properties
"ValidateExternalData" to False
it starts working.
I guess when your source is a variable it should be this way?

Robert
|||

Robert,

Could you please provide more details about your package? what is exactly variable on it? is the name of the table?

Phil is right, the number and data type of the columns cannot change in a dataflow. So, I guess that is not your case; but I am curious about your specific scenario.

|||I have a variable [SourceTable] that keeps a table name.
I am using it in my OLE DB Source Data Access Mode.
But with ValidateExternalMetadata=True it didn't work.
so I changed it to ValidateExternalMetadata=False
and it works now.

I don't know why Phil says it can not be done.
Even if I hold a table name in a variable
the bottom line is that all the tables have identical schema.

So as long as you set a default value of {SourceTable}
let's say to "Table1" it will work when
{SourceTable} changes to "Table2","Table3" and so on.

Robert
sql

Dynamic OLE DB Source and Destination

Hi,

I am building SSIS for 3 different files that have identical
schema and mapping logic.

In my OLE DB Source (object name - "OLEDBSource_SourceTable")
Data Access mode is "Variable name".
As soon as I swithced to this Data Acces mode
it started to give me an error:

[OLEDBSource_SourceTable [1]] Warning: The external metadata column collection is out of synchronization
with the data source columns.

The column "DEAL_NUM" needs to be updated in the external metadata column collection.
The "external metadata column "DEAL_NUM_Flag" (34529)" needs to be removed
from the external metadata column collection.
The "external metadata column "recordID" (33740)"
needs to be removed from the external metadata column collection.Meta data CANNOT change. You cannot setup an OLE DB source and have columns mapped and then just change the underlying source table. How can you imagine that working?

Search this forum for dynamic source/destinations and you'll get plenty of conversations on this topic.

Bottom line, SSIS relies on meta data. If you change it, how do you expect it to operate?|||

Hi Riga,

To make the data source dynamic you need to make the ConnectionString dynamic.

Hope this helps,

Andy

|||Everything turned out to be much easier, gentlemen!
As soon as I set in my OLE DB Source Properties
"ValidateExternalData" to False
it starts working.
I guess when your source is a variable it should be this way?

Robert|||

Robert,

Could you please provide more details about your package? what is exactly variable on it? is the name of the table?

Phil is right, the number and data type of the columns cannot change in a dataflow. So, I guess that is not your case; but I am curious about your specific scenario.

|||I have a variable [SourceTable] that keeps a table name.
I am using it in my OLE DB Source Data Access Mode.
But with ValidateExternalMetadata=True it didn't work.
so I changed it to ValidateExternalMetadata=False
and it works now.

I don't know why Phil says it can not be done.
Even if I hold a table name in a variable
the bottom line is that all the tables have identical schema.

So as long as you set a default value of {SourceTable}
let's say to "Table1" it will work when
{SourceTable} changes to "Table2","Table3" and so on.

Robert

Dynamic Flat File Destination Name!

Hi,

I am trying to access from OLE DB source. And based on one of the columns, I need to write the data to a Flat File Destination.

For Example,

CustID, ProductID, Product Name, Product Description

Say I am going to write to a different Flat File for every product. So if there are 10 products in the data. There should be 10 Flat Files. Also the file name should include the Product Name And Product ID.

It is being done in a single Data Flow Task.

Right now the Property Expression for the File Name is which is not working)

Code Snippet

@.DestFolder + [Data Conversion].ProductID + @.TodaysDate + ".txt"

The ProductIDs are in the ascending order. Any help or guidance?

Thanks

-Leo

You can't do that in one data flow unless you want to split it out to a separate destination (and connection manager) for each product. That might be okay for three or four products, but not if you have a bunch.

What you should do instead is set up a ForEach loop based on a the results of a SELECT DISTINCT ProductId query. Then execute your data flow once for each ProductId, customizing both the Source query and the destination connection manager with your ProductId.
|||

Thanks for your thoughts. OK, can we achieve this i.e. first 1000 records from the query to file1 and then next 1000 to file2. Or vice versa i.e. Write all the records to a single file and then read 1000 in first iteration and next 1000 records in the next iteration and so on, without multiple reads of the whole destination file.

Thanks

--Leo

|||You might be able to use the Export Column transform. That transform writes out a file per row based on one column containing the filename and another column containing the data.|||

Correct me if I am wrong, Ted, but the Export Column transform is used when you have a binary field in your data flow that you want to persist to a file. That doesn't seem to be the case for Leo's data.

Leo, you can partition the file by rows, but what Jay is suggesting would be simpler and perform better. If you use a For Each Loop, you would get a list of all product IDs, then execute a data flow inside the loop for each product ID. Using expressions, you can alter both the OLE DB Source query and the destination file connection string for each iteration of the loop. You wouldn't ever read a destination file - you would only be writing rows to them. And you would only process each row once.

Here's a few examples of using ForEach loops:

http://sqljunkies.com/WebLog/knight_reign/archive/2005/03/25/9588.aspx

http://agilebi.com/cs/blogs/jwelch/archive/2007/03/21/using-for-each-to-iterate-a-resultset.aspx

http://www.sqlis.com/55.aspx

http://blogs.conchango.com/jamiethomson/archive/2005/05/30/1489.aspx

http://rafael-salas.blogspot.com/2007/02/ssis-loop-through-files-in-date-range.html

http://blogs.conchango.com/jamiethomson/archive/2005/06/15/SSIS_3A00_-Getting-a-value-out-of-a-file-to-use-it-in-our-package.aspx

and a search that might have some more:

http://search.live.com/results.aspx?q=foreach+&form=QBRE&q1=macro%3Ajamiet.ssis

|||The Export Column should work with text and ntext in addition to binary -- the column containing the file data just needs to be either a character or binary lob column. This is just another option for Leo with a different approach, requiring the data and file columns to be derived, which he might be able to do since he has already derived the filename column in his original post.|||

New Leo wrote:

Thanks for your thoughts. OK, can we achieve this i.e. first 1000 records from the query to file1 and then next 1000 to file2. Or vice versa i.e. Write all the records to a single file and then read 1000 in first iteration and next 1000 records in the next iteration and so on, without multiple reads of the whole destination file.

Thanks

--Leo

Leo,

Yes you can do this. I've covered this exact problem here:

Splitting a file into multiple files

(http://blogs.conchango.com/jamiethomson/archive/2005/12/04/SSIS-Nugget_3A00_-Splitting-a-file-into-multiple-files.aspx)

You won't be able to do it in a single data-flow tho.

-Jamie

Dynamic Database Connection

Hi,

I done some Crystal reports(10.0) in asp .net using SQL server 2000(DB). While creating the reports, OLE DB Connection Information is must. Now i want to change the Source and Database for all the Reports. I want to do it Dynamically. Could I ?. I want to create reports without giving these(Source and Database) all. That is for each reports I dont want to give Source,Database,User Id and Password.
Pls help me in this regard.almost same problem with mine..
but mine is in report environment in vb6.

i'm not sure, but in vb6 (it think its not quite different :) )
u can do such this ..
firstly u must make a reference to the crystal report.
next...
--create a new Crystal Report object ...
it may looks like this

dim cr as new crystal report object

i think there must be a method of that CR object that could add a new report

hope this gonna help at least give a clue.. since i'm not that expert in .NET tech ..

good luck|||Hi szpilman,
The Problem is while creating the reports we have to give the Source and Database, But we should not do in this method. While creating the reports we should point one thing( It may be DSN).For this , All the reports should point out one (DSN(applies for VB .NET)). Here I want to point out all the reports to one( like DSN). Whenever I want to Change i should change in this Pointed one( like DSN ). Like DSN "i want one", where i can change the Source and Database dynamically.
Thank U