BulkLoader Class

com.bea.content.loader.bulk
BulkLoader Class

public class BulkLoader

    extends Object
    implements FilenameFilter

The Content Manager bulk loader application.

This class will scan the local file system for files to load via the NodeOps and WorkspaceOps api.

BulkLoader has limited use when loading against a Library Services Enabled Repository. If the content is new (not currently loaded into the BEA Repository) and Library Services is enabled, the content may be loaded. In this case, the lifecyclestatus must be specified in the md.properties, which defines the status the content will be checked in as. All lifecycle actions will operate as if the bulkloader user is a user in the adminn tools. So if a content item is checked in as "Ready" then the assignements will occur. Please review the LifeCycle javadoc for valid statuses, status transitions and how the system operates for each specific transition. Valid transition values (integer values) for BulkLoader md.properties are as follows:

  • 1 = DRAFT
  • 2 = READY
  • 4 = PUBLISHED
  • 5 = RETIRED

    The type (ObjectClass) for the file along with the values of any required properties must be specified in the metata data properties file. Thus, a type must be defined in the content repository before the bulkloader may load files from the file system.

    A folder will be loaded as a Hierarchy Node and a file will be loaded as a Content Node.

    The actual bytes will be loaded into the primary property (must be defined in the type) and must be of type Binary.

    In order for bulkloader to run, the repository and the application must be passed in as arguments and the server must be running.

    This class is mainly designed to run as a command-line application, via a "java com.bea.content.loader.bulk.BulkLoader" command-line. To see a usage, give it a -h flag or read the Usage.txt in this package.

    Additionally, BulkLoader objects can be created and used to provide the functionality in other places. The lifecycle of a BulkLoader is as follows:

    The base directory that will be loaded may be passed in using the -d paramter. If it is not specified then the current directory "." will be used. Any additional argument will be considered a file/folder to load relative to the base directory, or if an absolute path is specified then it will be used.

    Not calling parseArgs() and validateArgs(), in that order, will cause the BulkLoader object to most likely ungracefully fail. However, once those methods have been invoked, doLoad(), can invoked.

    If manually constructing and utilizing a BulkLoader object, be certain to synchronize all access to the object. Since the command-line program is single-threaded, BulkLoader objects are not thread-safe by design.

    To load the default LoaderFilter, the BulkLoader looks for the com/bea/content/loader/bulk/loader.properties file in the CLASSPATH. From that it reads the list of default LoaderFilter class names from the "loader.defFilters" property. To not use any of the default filters, specify +filters in the command-line args.

    If BulkLoader fails with an out of memory error you should increase your java heap size. You may do this in the bulkloader script by passing -Xms???m into the java call where ??? is the number of megabytes.

    Related Topics

    MetaParser
    FileCache


    Hierarchy
    Object
      BulkLoader
    All Implemented Interfaces

    FilenameFilter

    Nested Class Summary

    public static classBulkLoader.ShowUsageException
               Quick inner exception thrown on parseArgs() to say we should just print a usage report.

    Field Summary

    public String
    baseDirectory
    The contentBase.
    protected String
    batchFileName
    The batch properties file with the user name and password properties
    protected String
    DEF_ENCODE_PREFIX
    Encode batch property prefix
    public static final String
    DEF_MD_FILE_EXT
    The default file extension for metadata property files.
    public static final String
    DEF_MIME_TYPE
    The default mime type.
    public static final String
    DEF_WLS_PROPS_PATH
    The default weblogic properties file path.
    protected String
    deletePath
    The path of the hierarchy to delete, starting with a "/".
    public boolean
    doMetaParse
    Are we supposed to parse '*.htm' and '*.html' files for META tags.
    public String
    fileEncoding
    The file enconding (null for VM default).
    public List
    fileList
    The list of files/directories to scan over.
    public List
    htmlMatchList
    The list of patterns that represent HTML file names.
    public boolean
    ignoreErrors
    Do we ignore errors.
    public List
    ignoreList
    The list of file name patterns to ignore.
    public boolean
    includeHidden
    Are we supposed to include hidden files and directories.
    public boolean
    inheritProps
    Are we supposed to inherit metadata properties when recursing directories?
    protected boolean
    isFileSystem
    Flag indicating whether or not this repository is a filesystem repository
    public static final String
    JNDI_FACTORY
    Defines the JNDI context factory.
    protected String
    jndiName
    The jndi home for the remote Loader session bean.
    public List
    loaderFilters
    The list of LoaderFilters to try.
    public List
    matchList
    The list of file name patterns to include.
    public String
    mdFileExt
    The file extension of metadata property files.
    protected Collection
    metadataNames
    The metadata properties we find along the way.
    protected long
    numDocsLoaded
    The number of nodes we've loaded so far.
    protected String
    password
    The password for the user of this resource.
    public boolean
    recurse
    Do we recurse over directories?
    protected String
    repository
    The Repository to run the BulkLoader against.
    protected String
    url
    The WLS instance host where the content manager is running.
    protected String
    user
    The user of this resource.
    public boolean
    verbose
    Do we spew out messages.
     

    Constructor Summary

    BulkLoader()

    Constructor a BulkLoader without command-line arguments.
    BulkLoader(String args)

    Construct a BulkLoader from the given command-line arguments.
     

    Method Summary

    public boolean
    accept(File dir, String name)
    Implement the FilenameFilter interface method to use our match and ignore lists.
    public boolean
    checkFileAttributes(File f)
    public void
    debug(String mesg)
    Out put a debug message.
    public void
    doDelete()
    Do the actual bulk load logic on the file list.
    public void
    doLoad()
    Do the actual bulk load logic on the file list.
    public void
    doLoad(File baseDir, String path, Properties mdProperties)
    Load the given path into the database.
    public void
    error(String mesg, Throwable ex)
    Output an error message.
    public void
    error(String mesg)
    Output an error message.
    public void
    finished()
    Once you are done remove the bean for cleanup
    public String
    fixPath(String path)
    Fix up a path to be forward-slash style and to not have empty path parts.
    public Properties
    getLoaderFilterProperties(File f, Properties p)
    Get the properties from the BulkLoader's LoaderFilters for the given file.
    public Properties
    getMetadataProperties(File base, Properties p)
    Get the metadata properties for the given file or directory.
    public void
    inspectCurrentDirectory(File f, String path, Properties mdProperties)
    public static boolean
    isHidden(File f)
    Check if the specified file is a hidden file.
    public boolean
    isHtmlFile(String name)
    Tell if the specified file name is an HTML file to the loader.
    public static boolean
    isReadableDirectory(String name)
    Check if the specified file name is a directory that we can get into.
    public void
    loadIndividualFile(File f, String path, Properties mdProperties)
    public static int
    main(BulkLoader loader, String[] args)
    The main method invoked on a BulkLoader instance.
    public static void
    main(String[] args)
    Command-line entry point.
    public void
    parseArgs(String[] args)
    Parse the given input arguments.
    public void
    printArgs()
    Prints the arguments as debug statements.
    public void
    processBatchProperties()
    public boolean
    shouldIgnore(String name)
    Tell if the loader should ignore the specified file name.
    public boolean
    shouldInclude(String name)
    Tell if the loader should include the specified file name.
    public void
    usage()
    Print the usage of the application.
    public void
    usage(PrintWriter out)
    Print the usage of the application.
    public void
    validateArgs()
    Validate that we have been passed correct arguments.
    public void
    warning(String mesg, Throwable ex)
    Output a warning message.
    public void
    warning(String mesg)
    Output a warning message.
     
    Methods from class java.lang.Object
    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
       
    Methods from interface java.io.FilenameFilter
    accept
     

    Field Detail

    baseDirectory

    public String baseDirectory
    
    The contentBase.


    batchFileName

    protected String batchFileName
    
    The batch properties file with the user name and password properties


    DEF_ENCODE_PREFIX

    protected String DEF_ENCODE_PREFIX
    
    Encode batch property prefix


    DEF_MD_FILE_EXT

    public static final String DEF_MD_FILE_EXT
    
    The default file extension for metadata property files.


    DEF_MIME_TYPE

    public static final String DEF_MIME_TYPE
    
    The default mime type.


    DEF_WLS_PROPS_PATH

    public static final String DEF_WLS_PROPS_PATH
    
    The default weblogic properties file path.


    deletePath

    protected String deletePath
    
    The path of the hierarchy to delete, starting with a "/". The repository must be specified in the -repository argument.


    doMetaParse

    public boolean doMetaParse
    
    Are we supposed to parse '*.htm' and '*.html' files for META tags.


    fileEncoding

    public String fileEncoding
    
    The file enconding (null for VM default).


    fileList

    public List fileList
    
    The list of files/directories to scan over.


    htmlMatchList

    public List htmlMatchList
    
    The list of patterns that represent HTML file names.


    ignoreErrors

    public boolean ignoreErrors
    
    Do we ignore errors.


    ignoreList

    public List ignoreList
    
    The list of file name patterns to ignore.


    includeHidden

    public boolean includeHidden
    
    Are we supposed to include hidden files and directories.


    inheritProps

    public boolean inheritProps
    
    Are we supposed to inherit metadata properties when recursing directories?


    isFileSystem

    protected boolean isFileSystem
    
    Flag indicating whether or not this repository is a filesystem repository


    JNDI_FACTORY

    public static final String JNDI_FACTORY
    
    Defines the JNDI context factory.


    jndiName

    protected String jndiName
    
    The jndi home for the remote Loader session bean.


    loaderFilters

    public List loaderFilters
    
    The list of LoaderFilters to try.


    matchList

    public List matchList
    
    The list of file name patterns to include.

    Empty to include all.


    mdFileExt

    public String mdFileExt
    
    The file extension of metadata property files.

    This should start with a ".".


    metadataNames

    protected Collection metadataNames
    
    The metadata properties we find along the way.


    numDocsLoaded

    protected long numDocsLoaded
    
    The number of nodes we've loaded so far.


    password

    protected String password
    
    The password for the user of this resource.


    recurse

    public boolean recurse
    
    Do we recurse over directories?


    repository

    protected String repository
    
    The Repository to run the BulkLoader against.


    url

    protected String url
    
    The WLS instance host where the content manager is running. Defaults to "t3://localhost:7001"


    user

    protected String user
    
    The user of this resource.


    verbose

    public boolean verbose
    
    Do we spew out messages.

     

    Constructor Detail

    BulkLoader

    public BulkLoader()
    
    Constructor a BulkLoader without command-line arguments.

    BulkLoader

    public BulkLoader(String[] args)
    
    Construct a BulkLoader from the given command-line arguments.

    Related Topics

    BulkLoader.parseArgs(String[])

     

    Method Detail

    accept(File, String) Method

    public boolean accept(File dir, 
                          String name)
    
    Implement the FilenameFilter interface method to use our match and ignore lists.


    checkFileAttributes(File) Method

    public boolean checkFileAttributes(File f)
    

    debug(String) Method

    public void debug(String mesg)
    
    Out put a debug message.

    Subclasses can override this method to change where messages go.


    doDelete() Method

    public void doDelete()
    throws Exception
    Do the actual bulk load logic on the file list.

    Exceptions

    Exception

    doLoad() Method

    public void doLoad()
    throws Exception
    Do the actual bulk load logic on the file list.

    Exceptions

    Exception

    doLoad(File, String, Properties) Method

    public void doLoad(File baseDir, 
                       String path, 
                       Properties mdProperties)
    throws Exception
    Load the given path into the database.

    If path is a directory, all files underneath it that match our patterns will be included. If path is a file, it will be loaded.

    Parameters

    baseDir
    the base directory (can be used to get absolute file paths).
    path
    the path to the file or directory (this can be multi-part, not just name).
    mdProperties
    the base md properties for file (this should be a clone this method can modify as needed).

    Exceptions

    Exception

    error(String, Throwable) Method

    public void error(String mesg, 
                      Throwable ex)
    
    Output an error message.

    Subclasses can override this method to change where messages go.


    error(String) Method

    public void error(String mesg)
    
    Output an error message.


    finished() Method

    public void finished()
    throws RemoteException, RemoveException, Exception
    Once you are done remove the bean for cleanup

    Exceptions

    RemoteException
    RemoveException
    Exception

    fixPath(String) Method

    public String fixPath(String path)
    
    Fix up a path to be forward-slash style and to not have empty path parts.


    getLoaderFilterProperties(File, Properties) Method

    public Properties getLoaderFilterProperties(File f, 
                                                Properties p)
    
    Get the properties from the BulkLoader's LoaderFilters for the given file.

    Parameters

    f
    the file.
    p
    the properties object to add to (null to create new one).

    Returns

    p.

    getMetadataProperties(File, Properties) Method

    public Properties getMetadataProperties(File base, 
                                            Properties p)
    throws IOException
    Get the metadata properties for the given file or directory.

    This does not do a META data parse.

    Parameters

    base
    the file or directory base path.
    p
    the properties to load into (null to create new).

    Returns

    the properties (p if p was not null).

    Exceptions

    IOException

    inspectCurrentDirectory(File, String, Properties) Method

    public void inspectCurrentDirectory(File f, 
                                        String path, 
                                        Properties mdProperties)
    throws Exception

    Exceptions

    Exception

    isHidden(File) Method

    public static boolean isHidden(File f)
    
    Check if the specified file is a hidden file.

    Under UNIX, the File.isHidden() reports that "/weblogicCommerce/dmsBase/." is a hidden file, which it is not. So, this fixes that problem by getting canonicals paths for directories before calling isHidden(). That seems to do the trick.


    isHtmlFile(String) Method

    public boolean isHtmlFile(String name)
    
    Tell if the specified file name is an HTML file to the loader.


    isReadableDirectory(String) Method

    public static boolean isReadableDirectory(String name)
    
    Check if the specified file name is a directory that we can get into.


    loadIndividualFile(File, String, Properties) Method

    public void loadIndividualFile(File f, 
                                   String path, 
                                   Properties mdProperties)
    throws Exception

    Exceptions

    Exception

    main(BulkLoader, String[]) Method

    public static int main(BulkLoader loader, 
                           String[] args)
    
    The main method invoked on a BulkLoader instance.

    This will take a BulkLoader through the bulk loading steps. Output will be sent via the BulkLoader's debug(), warning(), and error() methods.

    This will not call System.exit().

    Parameters

    loader
    the command-line args.

    Returns

    the exit code (0 for success, non-zero for failure).

    Related Topics

    BulkLoader.parseArgs(String[])
    BulkLoader.validateArgs()


    main(String[]) Method

    public static void main(String[] args)
    
    Command-line entry point.

    This will call System.exit() on invalid args or error. To invoke a bulk load from your own code, create and manipulate a BulkLoader object. You can use the other main method, which does not exit.

    Parameters

    args
    the command-line args.

    Related Topics

    BulkLoader.main(BulkLoader, String[])


    parseArgs(String[]) Method

    public void parseArgs(String[] args)
    throws IllegalArgumentException
    Parse the given input arguments.

    Parameters

    args
    the input arguments.

    Exceptions

    IllegalArgumentException

    printArgs() Method

    public void printArgs()
    
    Prints the arguments as debug statements.


    processBatchProperties() Method

    public void processBatchProperties()
    throws Exception

    Exceptions

    Exception

    shouldIgnore(String) Method

    public boolean shouldIgnore(String name)
    
    Tell if the loader should ignore the specified file name.


    shouldInclude(String) Method

    public boolean shouldInclude(String name)
    
    Tell if the loader should include the specified file name.


    usage() Method

    public void usage()
    
    Print the usage of the application.


    usage(PrintWriter) Method

    public void usage(PrintWriter out)
    
    Print the usage of the application.


    validateArgs() Method

    public void validateArgs()
    throws IllegalStateException
    Validate that we have been passed correct arguments.

    This does not validate that the arguments are valid. That will be done in initialize().

    Exceptions

    IllegalStateException

    warning(String, Throwable) Method

    public void warning(String mesg, 
                        Throwable ex)
    
    Output a warning message.

    Subclasses can override this method to change where messages go.


    warning(String) Method

    public void warning(String mesg)
    
    Output a warning message.