Migrating From Filenet to Content Manager OnDemand - Revision history

Jderrick: /* Name output files in AG.Date.App format */

2025-01-15T16:22:09Z

Name output files in AG.Date.App format

← Older revision		Revision as of 16:22, 15 January 2025
Line 115:		Line 115:
	Write your report files with the following prefix: ApplicationGroup.DateFormat.Application		Write your report files with the following prefix: ApplicationGroup.DateFormat.Application

	This will allow you to load by Application Group, Date, and Application easily when you sort the files. By having the date as your second 'field' in the file name, you make loading in Chronological order easy and straightforward.		This will allow you to load by Application Group, Date, and Application easily when you sort the files. By having the date as your second 'field' in the file name, you make loading in Chronological order easy and straightforward. Regardless of your locale, using the YYYY-MM-DD format (with leading zeros) is wise, as most UNIX/Linux operating systems will natively sort file lists (like ls, or wildcards like "*" -- but not the 'find' command which outputs data in the order they appear in the filesystems inodes) which optimizes the way database tables for each individual App Group (known as 'segment tables' in CMOD lingo) are created.

	=== Concatenate Reports ===		=== Concatenate Reports ===

Jderrick: A variety of small changes, addition of the paragraph about data validation.

2017-02-11T07:21:40Z

A variety of small changes, addition of the paragraph about data validation.

← Older revision		Revision as of 07:21, 11 February 2017
Line 45:		Line 45:

	OnDemand doesn't like to have indexes defined in the Application Group without a corresponding value appearing in the reports it processes -- it also wastes space inside the database. It seems common in the Filenet world to assign a report to a Document Class that has indexes configured that simply don't exist anywhere in the report. Yes, you can assign default values to the empty fields to get ACIF to stop complaining, but if you want to do this right, you'll want to look into your index usage. Not just which fields you're populating most often, but also which fields your end users are searching on. Eliminating unused fields from Application Group definitions will streamline indexing, reduce storage costs, and reduce complaints from end users at the end of the day.		OnDemand doesn't like to have indexes defined in the Application Group without a corresponding value appearing in the reports it processes -- it also wastes space inside the database. It seems common in the Filenet world to assign a report to a Document Class that has indexes configured that simply don't exist anywhere in the report. Yes, you can assign default values to the empty fields to get ACIF to stop complaining, but if you want to do this right, you'll want to look into your index usage. Not just which fields you're populating most often, but also which fields your end users are searching on. Eliminating unused fields from Application Group definitions will streamline indexing, reduce storage costs, and reduce complaints from end users at the end of the day.

			=== Perform quality checks on existing metadata ===

			Over the years, and substantial amount of junk can accumulate inside database tables. Bad data that passes a very rudimentary check at the time it was loaded starts accumulating in databases, and is often discovered during the migration process. A common example is a bad date value -- someone meant to type in a date for 2013, but entered 2031 instead, or there's a mix-up between the month and day fields. (February 12th as 02/12 vs. December 2nd as 12/02). There needs to be an additional 'sanity check' on data before performing the migration, to identify and correct these issues before importing them into the new IBM CMOD server.

	=== Check for field re-use ===		=== Check for field re-use ===
Line 52:		Line 56:
	=== Transfer in Original Formats ===		=== Transfer in Original Formats ===

	For some Filenet installations, upstream servers (or intermediate file transfer systems) convert report data (from EBCDIC to ASCII) and change the formatting of the report. Content Manager OnDemand doesn't need any data transformation, and can ingest EBCDIC reports (of fixed record length, stream, or variable record lengths) directly and without conversion. Some conversion tools (I'm looking at you, MQ Series File Transfer Edition) can be configured to change the report so drastically, that CMOD can't properly index it.		For some Filenet installations, upstream servers (or intermediate file transfer systems) convert report data (from EBCDIC to ASCII) and change the formatting of the report. IBM Content Manager OnDemand doesn't need any data transformation, and can ingest EBCDIC reports (of fixed record length, stream, or variable record lengths) directly and without conversion. Some conversion tools (I'm looking at you, MQ Series File Transfer Edition) can be configured to change the report so drastically, that IBM CMOD can't properly index it.

	Wherever possible, remove any data conversion and deliver report data to OnDemand in its original format.		Wherever possible, remove any data conversion and deliver report data to OnDemand in its original format.
Line 68:		Line 72:
	=== Annotations ===		=== Annotations ===

	If you're using in Filenet, you'll need to extract them, and integrate them into CMOD. Annotations can be added to individual documents in OnDemand at load time, through the Generic Index files.		If you're using in Filenet, you'll need to extract them, and integrate them into IBM CMOD. Annotations can be added to individual documents in OnDemand at load time, through the Generic Index files.

	=== Reconciliation ===		=== Reconciliation ===
	In order to reconcile the documents after the migration is complete, you'll want to have a unique identifier for each document that needs to be moved out of FileNet and into CMOD. Thankfully, FileNet provides a unique 'Document Indentifier' or "DocID" -- which can be loaded into CMOD by adding a corresponding field in the Application Group definition. The DocID field doesn't need to be added to ~~CMOD~~ folders, so it can remain invisible to end users, or exposed to a different folder to be used with existing tools that use them.		In order to reconcile the documents after the migration is complete, you'll want to have a unique identifier for each document that needs to be moved out of FileNet and into CMOD. Thankfully, FileNet provides a unique 'Document Indentifier' or "DocID" -- which can be loaded into IBM CMOD by adding a corresponding field in the Application Group definition. The DocID field doesn't need to be added to OnDemand folders, so it can remain invisible to end users, or exposed to a different folder to be used with existing tools that use them.

	=== Don't create unnecessary directories ===		=== Don't create unnecessary directories ===

Jderrick: Updated CMOD to IBM CMOD.

2017-02-11T07:09:39Z

Updated CMOD to IBM CMOD.

← Older revision		Revision as of 07:09, 11 February 2017
Line 40:		Line 40:
	=== Converting Document Classes ===		=== Converting Document Classes ===

	Content Manager OnDemand ("CMOD") has an entirely different architecture than the Filenet products. In CMOD vernacular, an 'Application' is analogous to an individual report. But in CMOD, the top of the hierarchy is the 'Application Group' -- a grouping of Applications (aka 'reports') where the index fields, storage, and retention requirements are all the same. Properly defined Application Groups can have multiple Applications (again, 'reports') that belong to it. The most rational way to design Application Groups is to combine reports together that fulfill a specific business need. Human Resources reports shouldn't intermingle with Accounts Payable (even if they have the same index fields), and are kept logically separate by keeping their reports in separate Application Groups.		IBM Content Manager OnDemand ("CMOD") has an entirely different architecture than the Filenet products. In IBM CMOD vernacular, an 'Application' is analogous to an individual report. But in CMOD, the top of the hierarchy is the 'Application Group' -- a grouping of Applications (aka 'reports') where the index fields, storage, and retention requirements are all the same. Properly defined Application Groups can have multiple Applications (again, 'reports') that belong to it. The most rational way to design Application Groups is to combine reports together that fulfill a specific business need. Human Resources reports shouldn't intermingle with Accounts Payable (even if they have the same index fields), and are kept logically separate by keeping their reports in separate Application Groups.

	=== Quantify index usage ===		=== Quantify index usage ===

Jderrick: Clarified Application definition.

2017-02-11T07:08:51Z

Clarified Application definition.

← Older revision		Revision as of 07:08, 11 February 2017
Line 29:		Line 29:

	; Application (aka "App")		; Application (aka "App")
	: Defines the type of document (AFP, Line data, PDF, Image), and how to collect ~~search criteria~~ (aka 'indexes') for storage in the database.		: Defines the type of document (AFP, Line data, PDF, Image), and how to automatically collect metadata (aka 'indexes') that will be used as search criteria by end users, for storage in the database.
	: Multiple Applications of any data type (ie, AFP for customer statements, Line data reports generated by a mainframe, special letters or notices in PDF format, and incoming faxes stores as TIFF images).		: Multiple Applications of any data type (ie, AFP for customer statements, Line data reports generated by a mainframe, special letters or notices in PDF format, and incoming faxes stores as TIFF images).

	; Folder		; Folder
	: The Folder in OnDemand abstracts the internal complexities of Application Groups and Applications, and presents users with the fields that they can search (which were populated into the database by the Application) and sets limits on their queries (maximum number of returned hits, fields required for searches, etc.)		: The Folder in OnDemand abstracts the internal complexities of Application Groups and Applications, and presents users with the fields that they can search (which were populated into the database by the Application) and sets limits on their queries (maximum number of returned hits, fields required for searches, etc.)


	== Considerations ==		== Considerations ==

Jderrick: Minor changes to the Filenet to IBM CMOD Migration

2017-02-11T07:06:13Z

Minor changes to the Filenet to IBM CMOD Migration

← Older revision		Revision as of 07:06, 11 February 2017
Line 11:		Line 11:
	</blockquote>		</blockquote>

	Large migration projects are always painful. They take more time & money than anticipated (or more importantly, budgeted) because of one simple, honest truth: You are re-living all of the mistakes, problems, errors, and previously undiscovered issues with potentially '''decades''' of information. A lot of the 'institutional knowledge' surrounding specific data, reports, or use-cases is long gone. Especially frustrating are the conversations that end in "So, why did you think they did ''THAT'' ?!?!?", because it's not always obvious what the situation was when critical decisions were made.		Large migration projects are always painful. They take more time & money than anticipated (or more importantly, budgeted) because of one simple, honest truth: You are re-living all of the mistakes, problems, errors, and previously undiscovered issues with potentially '''decades''' of information. A lot of the 'institutional knowledge' surrounding specific data, reports, or use-cases is long gone. Especially frustrating are the conversations that end in "So, why did you think they did ''THAT'' ?!?!?", because it's not always obvious what the situation was when critical (and possibly bad) decisions were made.

	Patience is a virtue, and any sufficiently large migration project will test your determination, and at times, your sanity. The advice below comes from one of the largest migration projects I've ever participated in.		Patience is a virtue, and any sufficiently large migration project will test your determination, and at times, your sanity. The advice below comes from one of the largest migration projects I've ever participated in.

Jderrick: Added a section on choosing the Load Expiration type.

2016-03-31T13:59:41Z

Added a section on choosing the Load Expiration type.

← Older revision		Revision as of 13:59, 31 March 2016
Line 99:		Line 99:
	=== Group reports in chronological order ===		=== Group reports in chronological order ===
	Due to the way table segmentation works in OnDemand, you'll want to load the data in chronological order. When you name files, consider including a date field in YYYY-MM-DD format, so it can be sorted numerically at load time. This ensures that when the production server goes live, that end users will get speedy and fast database queries.		Due to the way table segmentation works in OnDemand, you'll want to load the data in chronological order. When you name files, consider including a date field in YYYY-MM-DD format, so it can be sorted numerically at load time. This ensures that when the production server goes live, that end users will get speedy and fast database queries.

			=== Use Expiration Type of "Load" ===
			In Content Manager OnDemand, there are three options for expiring data loaded into Application Groups:
			;Document: examines each row of metadata to determine if it's eligible for expiration, and deletes individual rows from the database
			;Load: examines the arsload table to find loads in which ALL documents are eligible for expiration, and preforms an 'unload'
			;Segment: examines each database table (also know as a "Segment") to determine if ALL the documents in the table are eligible, then it simply drops the database table.

			For historical migrations, using the 'Load' expiration types, minimizing the date range inside a single load file, and loading in chronological order will prevent issues with the deletion or disposition of data in the future.

	=== Name output files in AG.Date.App format ===		=== Name output files in AG.Date.App format ===

Jderrick: Added quote, tweaked text.

2015-11-25T12:35:53Z

Added quote, tweaked text.

← Older revision		Revision as of 12:35, 25 November 2015
Line 5:		Line 5:

	=== A Warning ===		=== A Warning ===
	Large migration projects are always painful. They take more time than anticipated (or more importantly, budgeted) because of one simple, honest truth: You are re-living all of the mistakes, problems, errors, and ~~until-now~~ undiscovered issues with potentially '''decades''' of information. A lot of the 'institutional knowledge' surrounding specific data, reports, or use-cases is long gone. Especially frustrating are the ~~questions~~ that end in "So, why did you think they did ''THAT'' ?!?!?", because it's not always obvious what the situation was when critical decisions were made.
			<blockquote>
			"Abandon all hope, ye who enter here"
			-- Dante's Inferno
			</blockquote>

			Large migration projects are always painful. They take more time & money than anticipated (or more importantly, budgeted) because of one simple, honest truth: You are re-living all of the mistakes, problems, errors, and previously undiscovered issues with potentially '''decades''' of information. A lot of the 'institutional knowledge' surrounding specific data, reports, or use-cases is long gone. Especially frustrating are the conversations that end in "So, why did you think they did ''THAT'' ?!?!?", because it's not always obvious what the situation was when critical decisions were made.

	Patience is a virtue, and any sufficiently large migration project will test your determination, and at times, your sanity. The advice below comes from one of the largest migration projects I've ever participated in.		Patience is a virtue, and any sufficiently large migration project will test your determination, and at times, your sanity. The advice below comes from one of the largest migration projects I've ever participated in.

Jderrick: Added file naming convention.

2015-11-25T12:27:12Z

Added file naming convention.

← Older revision		Revision as of 12:27, 25 November 2015
Line 93:		Line 93:
	=== Group reports in chronological order ===		=== Group reports in chronological order ===
	Due to the way table segmentation works in OnDemand, you'll want to load the data in chronological order. When you name files, consider including a date field in YYYY-MM-DD format, so it can be sorted numerically at load time. This ensures that when the production server goes live, that end users will get speedy and fast database queries.		Due to the way table segmentation works in OnDemand, you'll want to load the data in chronological order. When you name files, consider including a date field in YYYY-MM-DD format, so it can be sorted numerically at load time. This ensures that when the production server goes live, that end users will get speedy and fast database queries.

			=== Name output files in AG.Date.App format ===

			Write your report files with the following prefix: ApplicationGroup.DateFormat.Application

			This will allow you to load by Application Group, Date, and Application easily when you sort the files. By having the date as your second 'field' in the file name, you make loading in Chronological order easy and straightforward.

	=== Concatenate Reports ===		=== Concatenate Reports ===

Jderrick: Added info on field-re-use.

2015-11-03T11:48:06Z

Added info on field-re-use.

← Older revision		Revision as of 11:48, 3 November 2015
Line 40:		Line 40:

	OnDemand doesn't like to have indexes defined in the Application Group without a corresponding value appearing in the reports it processes -- it also wastes space inside the database. It seems common in the Filenet world to assign a report to a Document Class that has indexes configured that simply don't exist anywhere in the report. Yes, you can assign default values to the empty fields to get ACIF to stop complaining, but if you want to do this right, you'll want to look into your index usage. Not just which fields you're populating most often, but also which fields your end users are searching on. Eliminating unused fields from Application Group definitions will streamline indexing, reduce storage costs, and reduce complaints from end users at the end of the day.		OnDemand doesn't like to have indexes defined in the Application Group without a corresponding value appearing in the reports it processes -- it also wastes space inside the database. It seems common in the Filenet world to assign a report to a Document Class that has indexes configured that simply don't exist anywhere in the report. Yes, you can assign default values to the empty fields to get ACIF to stop complaining, but if you want to do this right, you'll want to look into your index usage. Not just which fields you're populating most often, but also which fields your end users are searching on. Eliminating unused fields from Application Group definitions will streamline indexing, reduce storage costs, and reduce complaints from end users at the end of the day.

			=== Check for field re-use ===

			As is human nature, some shortcuts may have been taken in the distant past that are now your problem to resolve. One common annoyance is the 're-use' of a particular field for a new data type. The time saved in putting a value in a pre-existing field with an unrelated name is likely to be a recurring theme. For American companies, using Social Security Numbers ("SSNs") for customers or employee records used to be common. With the advent of large-scale identity fraud, companies ripped out SSNs from databases, and replaced them with different numbering systems. In order to save time and money, many companies simply re-used their old "Social" field with new "Employee" numbers, often causing chaos when new systems try and do any validation. Also, watch out for real metadata squeezed into generic fields, like 'Description' or 'Comment' or 'Notes'. You'll have to determine what exactly that data is, and where it truly belongs in order to complete the migration successfully.

	=== Transfer in Original Formats ===		=== Transfer in Original Formats ===

Jderrick: Added the warning.

2015-11-03T11:40:09Z

Added the warning.

← Older revision		Revision as of 11:40, 3 November 2015
Line 3:		Line 3:

	These are some recommendations and tips from my experiences migrating Filenet data to Content Manager OnDemand - since I'm not a Filenet admin, Filenet is outside the scope of this article -- this only deals with the OnDemand-specific tips and tricks to make your migration easier.		These are some recommendations and tips from my experiences migrating Filenet data to Content Manager OnDemand - since I'm not a Filenet admin, Filenet is outside the scope of this article -- this only deals with the OnDemand-specific tips and tricks to make your migration easier.

			=== A Warning ===
			Large migration projects are always painful. They take more time than anticipated (or more importantly, budgeted) because of one simple, honest truth: You are re-living all of the mistakes, problems, errors, and until-now undiscovered issues with potentially '''decades''' of information. A lot of the 'institutional knowledge' surrounding specific data, reports, or use-cases is long gone. Especially frustrating are the questions that end in "So, why did you think they did ''THAT'' ?!?!?", because it's not always obvious what the situation was when critical decisions were made.

			Patience is a virtue, and any sufficiently large migration project will test your determination, and at times, your sanity. The advice below comes from one of the largest migration projects I've ever participated in.

	== Nomenclature ==		== Nomenclature ==