5 Batch Loading Content

5.1.1 File Records

A batch load file is made up of file records, which are sets of name/value pairs that specify the action to perform, or the metadata for individual content items, or both.

• Each file record ends with an <<EOD>> (end of data) marker.

• A pound sign (#) followed by a space at the beginning of a line indicates a comment. The comment character must be followed by a space. For example: # primaryFile=test.txt works properly, but #primaryFile=test.txt will cause errors.

• The following is an example of a file record:

  # This is a comment
  Action=insert
  dDocName=Sample1
  dDocType=Document
  dDocTitle=Batch Load record insert example
  dDocAuthor=sysadmin
  dSecurityGroup=Public
  primaryFile=links.doc
  dInDate=8/15/2001
  <<EOD>>
  

5.1.2 Actions

Valid actions for batch loading are Insert, Delete, and Update.

• If no action is specified for a file, the system tries to perform an update.

• Each file record can have only one action, but file records with different actions can be present in the same batch load file.

• The logic process for each action is different.

5.1.3 Insert

The insert action checks a new file in to the Content Server repository. Figure 5-1 illustrates the insert action.

• If the Content ID (dDocName) does not exist in the Content Server database, then a new file is created.

• If the Content ID (dDocName) exists in the Content Server database, and no revision (dRevLabel) is specified, then a new revision is created.

• If the Content ID (dDocName) and the specified revision (dRevLabel) exist in the Content Server database, then no action is performed.

Figure 5-1 The Insert Action Sequence for Checking In a New File

Description of "Figure 5-1 The Insert Action Sequence for Checking In a New File"

5.1.4 Delete

The delete action deletes one or all revisions of an existing file from the Content Server repository. If the specified Content ID (dDocName) does not exist in the Content Server database, no action is performed. Figure 5-2 illustrates the delete action.

Figure 5-2 The Delete Action Sequence

Description of "Figure 5-2 The Delete Action Sequence"

Action=delete
dDocName=HR001
<<EOD>>
Action=delete
dDocName=HR002
dRevLabel=2
<<EOD>>

5.1.5 Update

The update action updates existing content items. One of the following actions occurs, depending on what items are present in the file record and what content exists in the system:

• A new revision of an existing content item is created.

• An existing file's metadata is updated.

• A new content item is inserted (Action=insert is performed).

  Note:

  Batch loaded revisions will not enter a workflow even if they meet the criteria for an active workflow.

A new revision is created when one of the following scenarios occur:

Figure 5-3 The Update Action Sequence

Description of "Figure 5-3 The Update Action Sequence"

Action=update
dDocName=HR001
dDocType=Form
dDocTitle=New Employee Form
dDocAuthor=Olson
dSecurityGroup=Public
primaryFile=hr001.doc
DInDate=2/20/99
<<EOD>>
dDocName=HR002
dDocTitle=Payroll Change Form
primaryFile=hr002.doc
DIndate=2/20/99
dRevLabel=3
<<EOD>>

Action=update
dDocName=HR003
dDocType=Form
dDocTitle=Performance Review Template
dDocAuthor=Smith
primaryFile=hr003.doc
dIndate=2/20/99
dRevLabel=1
<<EOD>>

5.1.6 Optional Parameters

The following table lists the optional parameters you can use in any file record in a batch load file.

In a batch load file, there are two methods you can use to override the primary and alternate formats assigned to a content item check-in:

• Specifying a value for the primaryFile:format parameter, or specifying a value for the alternateFile:format parameter, both. However, it is possible to override these values by using the primaryOverrideFormat or alternateOverrideFormat parameters. It is also possible that certain components will force specific formats on certain types of check-ins or certain application functionality may exist in some components that forces a different format.

• Specifying a value for the primaryOverrideFormat parameter, or specifying a value for the alternateOverrideFormat parameter, or both. However, these will only work as parameters in the batch load file if you enable the IsOverrideFormat configuration variable. Note that using this method will override any values that you set for the primaryFile:format and alternateFile:format parameters.

  Optional Parameters	Definition
  dRevLabel	The metadata field named Revision. Maximum field length is 10 characters. Values must be an integer or comply with the Major/Minor Revision Label Sequence established by the System Properties settings.
  dDocAccount	The metadata field named Accounts. Maximum field length is 30 characters. This field is not carried over to the next file record. Do not specify this field if accounts are not enabled. If accounts are enabled and this field is not specified, dDocAccount will be set to an empty value.
  xComments	The metadata field named Comments. Maximum field length is 255 characters.
  dOutDate	The metadata field named Expiration Date. The dOutDate must use the date format of the locale of the user executing the Batch Loader. For example, the English-US date format is mm/dd/yy hh:mm:ss am/pm. Time information is optional. If you specify the time, only the hh:mm part is required. The ss and am/pm parts are optional.
  primaryFile:path	Specifies the location of the file. If a primaryFile:path value is specified, the value overrides the value specified for the primaryFile parameter. However, the primaryFile:path value is not used to determine the file conversion format. If a value for primaryFile:path is not specified, the location is determined from the primaryFile value. This parameter uses the following syntax: primaryFile:path=complete_path/primary_file_name
  primaryFile:format	Specifies the file format to use for the Primary File. This file format overrides the one specified by the file extension of the file and the value specified for the primaryFile parameter. If a primaryFile:format value is not specified, the file format is determined from the file extension for the primaryFile value. This parameter uses the following syntax: primaryFile:format=application/conversion_type
  alternateFile	The metadata field named Alternate File. The Alternate File name can be a complete path or just the file name. If a file name only is specified, the location of the file is determined as follows: If the SetFileDir optional parameter has been set in this file record or any previous file record, the directory specified in SetFileDir will be used. If the SetFileDir parameter has not been set, the batch load file path is used. (The path is specified in the Batch Load File field on the Batch Loader window.)
  alternateFile:path	Specifies the location of the alternate file. If an alternateFile:path value is specified, the value overrides the value specified for the alternateFile parameter. However, the alternateFile:path value is not used to determine the file conversion format. If an alternateFile:path value is not specified, the location is determined from the alternateFile parameter, if a value is specified. Otherwise, by default, the primaryFile value is used for the computation. This parameter uses the following syntax: alternateFile:path=complete_path
  alternateFile:format	Specifies the file format to use for the Alternate File. This file format overrides the one specified by the file extension of the file and the value specified for the alternateFile parameter. If an alternateFile:format value is not specified, the file format is determined from the file extension for the alternateFile parameter, if a value is specified. Otherwise, by default, the primaryFile value is used for the computation. This parameter uses the following syntax: alternateFile:format=application/conversion_type
  webViewableFile	The webViewableFile name can be a complete path or just the file name. If a webViewableFile value is specified, then the conversion process is not performed. If a file name only is specified, the location of the file is determined as follows: If the SetFileDir optional parameter has been set in this file record or any previous file record, the directory specified in SetFileDir will be used. If the SetFileDir parameter has not been set, the batch load file path is used. (The path is specified in the Batch Load File field on the Batch Loader window.)
  webViewableFile:path	Specifies the location of the web viewable file. If a webViewableFile.path value is specified, the value overrides the value specified for the webViewableFile parameter. However, the webViewableFile:path value is not used to determine the file conversion format. If a webViewableFile:path value is not specified, the location is determined from the webViewableFile parameter, if a value is specified. Otherwise, by default, the primaryFile value is used for the computation. This parameter uses the following syntax: webViewableFile:path=complete_path
  webViewableFile:format	Specifies the file format to use for the web viewable file. This file format overrides the one specified by the file extension of the file and the value specified for the webViewableFile parameter. If a webViewableFile:format value is not specified, the file format is determined from the file extension for the webViewableFile parameter, if a value is specified. Otherwise, by default, the primaryFile value is used for the computation. This parameter uses the following syntax: alternateFile:format=application/conversion_type
  primaryOverrideFormat	Specifies which file format to use for the Primary File. This file format overrides the one specified by the file extension of the file. This option will only work as a parameter if you enable the IsOverrideFormat configuration variable. You can set this variable by selecting the Allow Override Format in the System Properties application. However, a better (and recommended) alternative would be to use the primaryFile:format parameter.
  alternateOverrideFormat	Specifies which file format to use for the Alternate File. This file format overrides the one specified by the file extension of the file. This option will only work as a parameter if you enable the IsOverrideFormat configuration variable. You can set this variable by selecting the Allow Override Format in the System Properties application. However, a better (and recommended) alternative would be to use the alternate File:format parameter.
  SetFileDir	Specifies the directory where the Primary Files and Alternate Files are located. This field is carried over to the next file record.

  
  

5.1.7 Custom Metadata Fields

Any custom metadata field that has been defined in the Configuration Manager can be included in a file record.

• If you have defined any custom metadata fields as required fields, those fields must be defined for an insert action or an update action.

• If a custom metadata field is not a required field, but it has a default value (even if blank), then the default value will be used if the value is not specified in the batch load file.

• When specifying a custom metadata field value, the field name preceded with an x. For example, if you have a custom metadata field called Location, then the batch load file entry will be xLocation=value.

• Keep in mind that some add-on products use custom metadata fields. For example, if you have PDF Watermark, you will have created a field called Watermark. To include this field in a batch load file, precede it with an x just like any other custom metadata field (for example, xWatermark).

5.2.1 About Preparing a Batch Load File

You can use any method you prefer to create a batch load file, if the resulting text file conforms to the batch load file syntax requirements. However, the Batch Loader provides a tool called the BatchBuilder to assist you in creating batch load files.

• The BatchBuilder creates a batch load file based on the files in a specified directory. The BatchBuilder reads recursively through all the sub-directories to create the batch load file.

• A mapping file tells the BatchBuilder how to determine the metadata for each file record. You can use the BatchBuilder to create and save custom Mapping Files.

• You can run the BatchBuilder from the standalone application interface or from the command line.

• The BatchBuilder can also be used to create external collections of content, which are indexed and stored in a separate search collection rather than in the Content Server database. You can set up read-only external collections, where users can search for content but cannot update metadata or delete content. This option is recommended when external content is also included in another Content Server instance.

If you plan to use the Batch Loader utility to update and insert a large number of files on your Content Server instance simultaneously, you must create a batch load file. Two of the optional parameters that you can include in your batch load file are primaryOverrideFormat and alternateOverrideFormat. However, these options only work as parameters in the batch load file if you enable the IsOverrideFormat configuration variable. You can set this variable using the System Properties application.

5.2.2 Mapping Files

Mapping files are text files that have a .hda extension, which identifies them as a type of data file used by the Content Server instance.

For more information on HDA files, LocalData properties, and ResultSets, see Oracle Fusion Middleware Developing with Oracle WebCenter Content.

dDocType=<$dir1$>
dSecurityGroup=<$dir2$>
dDocAccount=<$dir3$>

<$dir1$> = "docs"
<$dir2$> = "public"
<$dir3$> = "sales"

dDocAuthor=<$dUser$>

dDocTitle=<$filename$>.<$extension$>

dDocName=<$filename$>

xPath=<$filepath$>

xFileSize=<$filesize$>

dInDate=<$filetimestamp$>

5.2.3 Creating a Batch Load File from the BatchBuilder Window

To create a batch load file from the BatchBuilder window:

1. Start the Batch Loader utility:

   • Windows: Choose Start, then Programs, then Content Server, then instance_name, then Utilities, then BatchLoader.

   • UNIX: Go to the DomainHome/ucm/cs/bin/ directory, type ./BatchLoader in a shell window, and press the RETURN key on your keyboard.

2. In the login window, enter the Content Server administrator user name and password, then click OK.

3. In the Batch Loader window, choose Options, then Build Batch File.

4. In the Directory field on the BatchBuilder window, enter the location of the files to be included in the batch load file.

5. In the Batch Load File field, enter the path and file name for the batch load file. You can click the Browse button to navigate to and select the directory and file.

6. From the Mapping list, select a mapping file. To create a new mapping file or edit an existing one, see Section 5.2.4.

7. Optional: In the File Filter field, enter filter settings to include or exclude particular files from the batch load file.

8. Optional: To batch load a read-only external collection, choose External, and select the external collection options.

9. Click Build.

10. When the build process is complete, click OK.

11. Open the batch load file in a text editor and double-check the file records.

12. To save the current batch load file settings as the default, choose Options, then Save Configuration.

5.2.4 Creating a Mapping File

To create a mapping file.

1. Display the BatchBuilder window.

2. Click Edit next to the Mapping field.

3. In the BatchBuilder Mapping List window, click Add.

4. In the Add BatchBuilder Mapping window, enter a name and description for the mapping file, and click OK.

5. In the Edit BatchBuider Mapping window, click Add.

6. In the Add/Edit BatchBuilder Mapping Field window, enter a metadata field name to be defined. For example, enter dDocName for the Content ID field, or xComments for the Comments field.

7. Enter the value for the metadata field.

   • Type any constant text and Idoc script directly in the Value field. For example, to set Document as the Type for all documents in the batch load file, enter dDocType in the Field field, and enter Document in the Value field. For more information, see Oracle Fusion Middleware Developing with Oracle WebCenter Content.

   • To add a predefined variable to the Value field, select the variable in the right column and click the << button. For example, to set each document's second-level directory as the Security Group, enter dSecurityGroup in the Field field, and insert the <$dir1$> variable in the Value field.

     Caution:

     Be careful when choosing predefined variables. Many metadata fields have length limitations and cannot contain certain characters (such as spaces or punctuation marks). See "Managing Content" in the Oracle Fusion Middleware Managing Oracle WebCenter Content.

8. Click OK.

9. Repeat steps 4 through 8 for as many metadata fields as you want to define.

10. Click OK to save changes and close the Edit BatchBuilder Mapping window.

    The mapping file is saved as MapFileName.hda in the IntradocDir/search/external/mapping/ directory.

11. Click Close to close the BatchBuilder Mapping List window.

5.2.5 Creating a Batch Load File from the Command Line

You can create a batch load file by entering the BatchBuilder parameters from a command line rather than entering them in the BatchBuilder window. To create a batch load file from the command line:

1. Open the DomainHome/ucm/cs/bin/intradoc.cfg file in a text editor, and add the following line, where sysadmin is the user name of the Content Server system administrator:

   BatchLoaderUserName=sysadmin
   

   This is required so that the system logs in as the system administrator, because only users who have admin rights have permission to run the Batch Loader and BatchBuilder applications.

2. Save and close the file.

3. Open a command line window and change to the DomainHome/ucm/cs/bin/ directory.

   Caution:

   Run the BatchBuilder using the same operating system account that runs the Content Server instance. Otherwise, the software might not process your data due to permissions problems.

4. Enter the following command:

   • Windows:

     BatchLoader.exe -spider -q -ddirectory -mmappingfile -nbatchloadfile
     

   • UNIX:

     BatchLoader -spider -q -ddirectory -mmappingfile -nbatchloadfile
     

The following flags can be used with the BatchLoader command to run the BatchBuilder from the command line:

BatchLoader.exe -spider -q -dc:/myfiles -mMyMappingFile -nc:/batching/batchinsert.txt -eexe,zip

BatchLoader -spider -q -d/myfiles -mMyMappingFile -n/batching/batchinsert.txt -eindex.htm,index.html

5.3.1 About Running the Batch Loader

The Batch Loader uses the information from a batch load file to check in (insert), delete, or update a large number of files on your Content Server instance simultaneously.

• You can run the Batch Loader from the standalone application interface or from the command line.

• After you run the Batch Loader, the Content Server instance processes files through the Inbound Refinery and the Indexer as it would for any other content item.

5.3.2 Batch Loading from the Batch Loader Window

To batch load content using the Batch Loader window:

1. Display the Batch Loader window.

2. Click Browse, navigate to and select the batch load file.

3. To change the number of errors that can occur before the Batch Loader stops processing, enter the number in the Maximum errors allowed field.

4. To delete files from the hard drive after they are successfully checked in or updated, select Clean up files after successful check in.

5. To create a text file containing the file records that failed during batch loading, select Enable error file for failed revision classes.

6. Click Load Batch File to start the Batch Loader process.

   When the batch load process is complete, a Batch Loader message window is displayed, indicating the number of errors that occurred, if any.

7. If you enabled the error file, write down the file name shown in the message box.

8. Click OK.

9. Correct any problems with the batch load.

10. To save the current Batch Loader settings as the default, choose Options, then Save Configuration.

5.3.3 Batch Loading from the Command Line

You can batch load content by entering the Batch Loader parameters from a command line rather than entering them in the Batch Loader window. To run the Batch Loader from the command line:

1. Open the DomainHome/ucm/cs/bin/intradoc.cfg file in a text editor, and add the following line, where sysadmin is the user name of the Content Server system administrator:

   BatchLoaderUserName=sysadmin
   

   This is required so that the system logs in as the system administrator, because only users who have admin rights have permission to run the Batch Loader application.

2. Save and close the file.

3. Open a command line window and go to the DomainHome/ucm/cs/bin/ directory.

   Caution:

   Run the Batch Loader using the same operating system account that runs the Content Server instance. Otherwise, the software might not process your files due to permissions problems.

4. Enter the following command:

   • Windows:

     BatchLoader.exe -q -nbatchloadfile
     

   • UNIX:

     BatchLoader -q -nbatchloadfile
     

   The Batch Loader processes the batch load file, but message boxes will not be shown.

5. Correct any problems with the batch load.

The following flags can be used with the BatchLoader command from the command line:

BatchLoader.exe -q -nc:/batching/batchinsert.txt

BatchLoader -q -n/batching/batchinsert.txt

5.3.4 Using the IdcCommand Utility and Remote Access

Occasionally, you may need to use remote access when managing your Content Server instance. This does not necessarily mean that remote terminal access is required. However, you must have the ability to submit commands to the server from a remote location.

Combining remote access with the IdcCommand utility provides a powerful toolset and an easy way to check in a large number of files to your instance. To take advantage of this functionality, you will need to properly set up the workstation to submit commands and be able to use the IdcCommand utility with a batch load command file.

This section covers the following topics:

• Section 5.3.4.1, "Batch Load Command Files"

• Section 5.3.4.2, "Preparing for Remote Batch Loading"

@Properties LocalData
IdcService=CHECKIN_UNIVERSAL
doFileCopy=1
dDocTitle=thisfile
dDocType=Native
dSecurityGroup=Internal
dDocAuthor=sysadmin
primaryFile=filename
primaryFile:Path=pathtothefile/primaryfilename
xComments=Initial Check In
@end
<<EOD>>@Properties LocalData
IdcService=CHECKIN_UNIVERSAL
doFileCopy=1
dDocTitle=99.tif
dDocType=Native
dSecurityGroup=Internal
dDocAuthor=sysadmin
primaryFile=350.afp
primaryFile:path=/lofs/invoices/350.afp
xComments=Initial Check In
@end
<<EOD>>

5.3.5 Batch Loading Content as Metadata Only

Depending on the action you plan to perform using the Batch Loader, certain fields are required in the batch load file. If you are updating only the metadata in existing content items, the primaryFile field is not required in the batch load file; for more information see Section 5.1.5.1.

However, if you want to load (insert action) content in to the Content Server instance as metadata only, then the primaryFile field is required in the batch load file. Although the field is ignored by the import, the Batch Loader expects it to be defined. If the primaryFile field is missing, you will get an error as follows (or similar):

Please check record number <number>. BatchLoader: unable to check in '<record>' because the required field 'primaryFile' is missing.

To batch load content as metadata only:

1. Open the Content Server instance config.cfg file:

   IntradocDir/config/config.cfg

2. Add the following configuration variables:

   createPrimaryMetaFile=true
   AllowPrimaryMetaFile=true
   

3. Save and close the config.cfg file.

4. In the batch load file, add the following fields for each record:

   primaryFile=
   createPrimaryMetaFile=true
   

   Note that leaving the primaryFile field blank is acceptable. The field is ignored but must be included.

5. Continue to batch load your content using the Batch Loader procedure or the command line procedure. For more information, see Section 5.3.2 or Section 5.3.3.

5.3.6 Batch Loader -console Command Line Switch

Adding the -console switch to the Batch Loader command line causes all output to be echoed to the HTML Content Server log and to the console window that is running the Batch Loader. Alternatively, you can use operating system redirects to send the output to a separate log file.

Command Line Example

• Windows command line:

  BatchLoader.exe -console -q -nc:/batching/batchinsert.txt
  

• UNIX command line:

  BatchLoader -console -q -n/u2/apps/batching/batchinsert.txt
  

Sample Output

Processed 1 of 4 record.
Processed 2 of 4 records.
Processed 3 of 4 records.
Processed 4 of 4 records.
Done processing batch file 'c:/batching/batchinsert.txt'. Out of 4 records processed, 4 succeeded and 0 errors occurred.

5.3.7 Adding a Redirect

You can use a redirect symbol on the command line to send the Batch Loader output to a separate log file. The symbol works on both UNIX and Windows. By default, the -console switch sends the Batch Loader's output to stderr. To redirect the output to a different file, use the special redirect symbol 2>.

In the following examples, each command must be entered all on one line.

• Windows command line with redirect:

  BatchLoader.exe -console -q -nc:/batching/batchinsert.txt 2> batchlog.txt
  

• UNIX command line with redirect:

  BatchLoader -console -q -n/u2/apps/batching/batchinsert.txt 2>
  /logs/CSbatchload.log
  

5.3.8 Correcting Batch Load Errors

To correct any errors that occur during batch loading.

1. Choose Administration, then Log Files, then Content Server Logs.

2. In the Content Server log file, look through the Type column for the word Error.

3. Read the description to determine the problem.

4. Fix the error in one of these files:

   • Batch load file

   • The error file for the failed content. (This option is available only if you enabled it on the Batch Loader window.) The error file is located in the same directory as the batch load file, with several digits appended to the batch load file name.

     Tip:

     If you rerun an entire batch load file, content items that have already been checked in will usually fail. This occurs because the release dates of the existing content items will be the same as the ones you are trying to insert.

Figure 5-4 Content Server log file

Description of "Figure 5-4 Content Server log file"

5.5.1 Background Information

A user wanted to load 27,000 content items in to the Content Server instance that was running on an AIX server. The DB2 database was running on a separate AIX server. The content items included TIFs as the native files and corresponding PDFs as the web-viewable files. Inbound Refinery generated thumbnails from the native files.

Initially during the batch load, the performance was acceptable with sub-second insert times. However, after a few thousand content items were loaded, the performance began to degrade. Content items started to require a few seconds to load and, eventually, the load time was over 10 seconds per content item.

5.5.2 Preliminary Troubleshooting

While the batch load was running, nothing seemed to be wrong with the Content Server instance. It had sufficient memory, the CPU utilization was low (less than 5%), and there were no disk bottlenecks. The Inbound Refinery server was busy, but was processing thumbnails at an acceptable rate.

Two issues were found with the database server:

• Two processes were taking turns to update the database. While one process was executing, the second process waited for other process to release database locks. When the first process completed, the second process executed while the first process waited. The processes in this execute/wait cycle included:

  • The actual batch load process that was updating the database tables after inserting a content item.

  • The Content Server instance was updating the database tables; changing the status from GENWWW to DONE after receiving notification that a thumbnail had been completed.

  The two processes should not have been contending with each other because they were not updating the same content items. It seemed that the two processes were locking each other out because DB2 had performed lock escalation and was now locking entire database pages instead of single rows.

• There were a large number of tablespace scans being performed by both processes.

5.5.3 Solution

A two-step solution was used:

1. Inbound Refinery was shut down to prevent the status update process from competing with the batch loading process. The performance did improve because there was a 2000+ backlog of content items from the completed thumbnails.

2. A RUNSTATS command was issued on all the Content Server database tables to update the table statistics. This dramatically improved the performance of the batch load. The insert time returned to sub-second and the batch load completed within a short amount of time. It took 21 hours to insert the first 22,000 content items. After updating the table statistics, the remaining 5,000 content items were inserted in 13 minutes.