Oracle Big Data Spatial and Graph on Apache Hadoop is a framework that uses the MapReduce programs and analytic capabilities in a Hadoop cluster to store, access, and analyze the spatial data. The spatial features provide a schema and functions that facilitate the storage, retrieval, update, and query of collections of spatial data. Big Data Spatial and Graph on Hadoop supports storing and processing spatial images, which could be geometric shapes, raster, or vector images and stored in one of the several hundred supported formats.

The advantages of using Oracle Big Data Spatial and Graph include the following:

• Unlike some of the GIS-centric spatial processing systems and engines, Oracle Big Data Spatial and Graph is capable of processing both structured and unstructured spatial information.

• Customers are not forced or restricted to store only one particular form of data in their environment. They can have their data stored both as a spatial or nonspatial business data and still can use Oracle Big Data to do their spatial processing.

• This is a framework, and therefore customers can use the available APIs to custom-build their applications or operations.

• Oracle Big Data Spatial can process both vector and raster types of information and images.

The spatial data is loaded for query and analysis by the Spatial Server and the images are stored and processed by an Image Processing Framework. You can use the Oracle Big Data Spatial and Graph server on Hadoop for:

• Cataloguing the geospatial information, such as geographical map-based footprints, availability of resources in a geography, and so on.

• Topological processing to calculate distance operations, such as nearest neighbor in a map location.

• Categorization to build hierarchical maps of geographies and enrich the map by creating demographic associations within the map elements.

The following functions are built into Oracle Big Data Spatial and Graph:

• Indexing function for faster retrieval of the spatial data.

• Map function to display map-based footprints.

• Zoom function to zoom-in and zoom-out specific geographical regions.

• Mosaic and Group function to group a set of image files for processing to create a mosaic or subset operations.

• Cartesian and geodetic coordinate functions to represent the spatial data in one of these coordinate systems.

• Hierarchical function that builds and relates geometric hierarchy, such as country, state, city, postal code, and so on. This function can process the input data in the form of documents or latitude/longitude coordinates.

The stored spatial data or images can be in one of these supported formats:

• GeoJSON files

• Shapefiles

• Both Geodetic and Cartesian data

• Other GDAL supported formats

You must have the following software, to store and process the spatial data:

• Java runtime

• GCC Compiler - Only when the GDAL-supported formats are used

For processing the raster data, the GDAL loader loads the raster spatial data or images onto a HDFS environment. The following basic operations can be performed on a raster spatial data:

• Mosaic: Combine multiple raster images to create a single mosaic image.

• Subset: Perform subset operations on individual images.

• Raster algebra operations: Perform algebra operations on every pixel in the rasters (for example, add, divide, multiply, log, pow, sine, sinh, and acos).

• User-specified processing: Raster processing is based on the classes that user sets to be executed in mapping and reducing phases.

This feature supports a MapReduce framework for raster analysis operations. The users have the ability to custom-build their own raster operations, such as performing an algebraic function on a raster data and so on. For example, calculate the slope at each base of a digital elevation model or a 3D representation of a spatial surface, such as a terrain. For details, see Oracle Big Data Spatial Hadoop Image Processing Framework for Raster Data Processing.

This feature supports the processing of spatial vector data:

• Loaded and stored on to a Hadoop HDFS environment

• Stored either as Cartesian or geodetic data

The stored spatial vector data can be used for performing the following query operations and more:

• Point-in-polygon

• Distance calculation

• Anyinteract

• Buffer creation

Sevetal data service operations are supported for the spatial vector data:

• Data enrichment

• Data categorization

• Spatial join

In addition, there is a limited Map Visualization API support for only the HTML5 format. You can access these APIs to create custom operations. For details, see "Oracle Big Data Spatial Vector Analysis."

The Image Loader is a Hadoop job that loads a specific image or a group of images into HDFS.

• While importing, the image is tiled and stored as an HDFS block.

• GDAL is used to tile the image.

• Each tile is loaded by a different mapper, so reading is parallel and faster.

• Each tile includes a certain number of overlapping bytes (user input), so that the tiles cover area from the adjacent tiles.

• A MapReduce job uses a mapper to load the information for each tile. There are 'n' number of mappers, depending on the number of tiles, image resolution and block size.

• A single reduce phase per image puts together all the information loaded by the mappers and stores the images into a special .ohif format, which contains the resolution, bands, offsets, and image data. This way the file offset containing each tile and the node location is known.

• Each tile contains information for every band. This is helpful when there is a need to process only a few tiles; then, only the corresponding blocks are loaded.

The following diagram represents an Image Loader process:

Description of the illustration image_loader_job.png

The Image Processor is a Hadoop job that filters tiles to be processed based on the user input and performs processing in parallel to create a new image.

• Processes specific tiles of the image identified by the user. You can identify one, zero, or multiple processing classes. These classes are executed in the mapping or reducing phase, depending on your configuration. For the mapping phase, after the execution of processing classes, a mosaic operation is performed to adapt the pixels to the final output format requested by the user. If no mosaic operation was requested, the input raster is sent to reduce phase as is. For reducer phase, all the tiles are put together into a GDAL data set that is input for user reduce processing class, where final output may be changed or analyzed according to user needs.

• A mapper loads the data corresponding to one tile, conserving data locality.

• Once the data is loaded, the mapper filters the bands requested by the user.

• Filtered information is processed and sent to each mapper in the reduce phase, where bytes are put together and a final processed image is stored into HDFS or regular File System depending on the user request.

The following diagram represents an Image Processor job:

Description of the illustration image_processor_job.png

The image loading job has its custom input format that splits the image into related image splits. The splits are calculated based on an algorithm that reads square blocks of the image covering a defined area, which is determined by

area = ((blockSize - metadata bytes) / number of bands) / bytes per pixel.

For those pieces that do not use the complete block size, the remaining bytes are refilled with zeros.

Splits are assigned to different mappers where every assigned tile is read using GDAL based on the ImageSplit information. As a result an ImageDataWritable instance is created and saved in the context.

The metadata set in the ImageDataWritable instance is used by the processing classes to set up the tiled image in order to manipulate and process it. Since the source images are read from multiple mappers, the load is performed in parallel and faster.

After the mappers finish reading, the reducer picks up the tiles from the context and puts them together to save the file into HDFS. A special reading process is required to read the image back.

The following input parameters are supplied to the Hadoop command:

hadoop jar /opt/oracle/oracle-spatial-graph/spatial/raster/jlib/hadoop-imageloader.jar 
  -files <SOURCE_IMGS_PATH>
  -out <HDFS_OUTPUT_FOLDER>
  -gdal <GDAL_LIB_PATH>
  -gdalData <GDAL_DATA_PATH>
  [-overlap <OVERLAPPING_PIXELS>]
  [-thumbnail <THUMBNAIL_PATH>]
  [-expand <false|true>]
  [-extractLogs <false|true>]
  [-logFilter <LINES_TO_INCLUDE_IN_LOG>]
  [-pyramid <OUTPUT_DIRECTORY, LEVEL, [RESAMPLING]>]

Where:

• SOURCE_IMGS_PATH is a path to the source image(s) or folder(s). For multiple inputs use a comma separator. This path must be accessible via NFS to all nodes in the cluster.
• HDFS_OUTPUT_FOLDER is the HDFS output folder where the loaded images are stored.
• OVERLAPPING_PIXELS is an optional number of overlapping pixels on the borders of each tile, if this parameter is not specified a default of two overlapping pixels is considered.
• GDAL_LIB_PATH is the path where GDAL libraries are located.
• GDAL_DATA_PATH is the path where GDAL data folder is located. This path must be accessible through NFS to all nodes in the cluster.
• THUMBNAIL_PATH is an optional path to store a thumbnail of the loaded image(s). This path must be accessible through NFS to all nodes in the cluster and must have write access permission for yarn users.
• -expand controls whether the HDFS path of the loaded raster expands the source path, including all directories. If you set this to false, the .ohif file is stored directly in the output directory (specified using the -o option) without including that directory’s path in the raster.
• -extractLogs controls whether the logs of the executed application should be extracted to the system temporary directory. By default, it is not enabled. The extraction does not include logs that are not part of Oracle Framework classes.
• -logFilter <LINES_TO_INCLUDE_IN_LOG> is a comma-separated String that lists all the patterns to include in the extracted logs, for example, to include custom processing classes packages.
• -pyramid <OUTPUT_DIRECTORY, LEVEL, [RESAMPLING]> allows the creation of pyramids while making the initial raster load. An OUPUT_DIRECTORY must be provided to store the local pyramids before uploading to HDFS; pyramids are loaded in the same HDFSA directory requested for load. A pyramid LEVEL must be provided to indicate how many pyramids are required for each raster. A RESAMPLING algorithm is optional to specify the method used to execute the resampling; if none is set, then BILINEAR is used.

For example, the following command loads all the georeferenced images under the images folder and adds an overlapping of 10 pixels on every border possible. The HDFS output folder is ohiftest and thumbnail of the loaded image are stored in the processtest folder.

hadoop jar /opt/oracle/oracle-spatial-graph/spatial/raster/jlib/hadoop-imageloader.jar   -files /opt/shareddir/spatial/demo/imageserver/images/hawaii.tif -out ohiftest -overlap 10 -thumbnail /opt/shareddir/spatial/processtest –gdal /opt/oracle/oracle-spatial-graph/spatial/raster/gdal/lib –gdalData /opt/shareddir/data

By default, the Mappers and Reducers are configured to get 2 GB of JVM, but users can override this settings or any other job configuration properties by adding an imagejob.prop properties file in the same folder location from where the command is being executed. This properties file may list all the configuration properties that you want to override. For example,

mapreduce.map.memory.mb=2560
mapreduce.reduce.memory.mb=2560
mapreduce.reduce.java.opts=-Xmx2684354560
mapreduce.map.java.opts=-Xmx2684354560

Java heap memory (java.opts properties) must be equal to or less than the total memory assigned to mappers and reducers (mapreduce.map.memory and mapreduce.reduce.memory). Thus, if you increase Java heap memory, you might also need to increase the memory for mappers and reducers.

For GDAL to work properly, the libraries must be available using $LD_LIBRARY_PATH. Make sure that the shared libraries path is set properly in your shell window before executing a job. For example:

export LD_LIBRARY_PATH=$ALLACCESSDIR/gdal/native

The reducer generates two output files per input image. The first one is the .ohif file that concentrates all the tiles for the source image, each tile may be processed as a separated instance by a processing mapper. Internally each tile is stored as a HDFS block, blocks are located in several nodes, one node may contain one or more blocks of a specific .ohif file. The .ohif file is stored in user specified folder with -out flag, under the /user/<USER_EXECUTING_JOB>/OUT_FOLDER/<PARENT_DIRECTORIES_OF_SOURCE_RASTER> if the flag –expand was not used. Otherwise, the .ohif file will be located at /user/<USER_EXECUTING_JOB>/OUT_FOLDER/, and the file can be identified as original_filename.ohif.

The second output is a related metadata file that lists all the pieces of the image and the coordinates that each one covers. The file is located in HDFS under the metadata location, and its name is hash generated using the name of the ohif file. This file is for Oracle internal use only, and lists important metadata of the source raster. Some example lines from a metadata file:

srid:26904
datatype:1
resolution:27.90809458890406,-27.90809458890406
file:/user/hdfs/ohiftest/opt/shareddir/spatial/data/rasters/hawaii.tif.ohif
bands:3
mbr:532488.7648166901,4303164.583549625,582723.3350767174,4269619.053853762
0,532488.7648166901,4303164.583549625,582723.3350767174,4269619.053853762
thumbnailpath:/opt/shareddir/spatial/thumb/

If the -thumbnail flag was specified, a thumbnail of the source image is stored in the related folder. This is a way to visualize a translation of the .ohif file. Job execution logs can be accessed using the command yarn logs -applicationId <applicationId>.

The image processing job has different flows depending on the type of processing requested by the user.

• Default Image Processing Job Flow: executed for processing that includes a mosaic operation, single raster operation, or basic multiple raster operation.

• Multiple Raster Image Processing Job Flow: executed for processing that includes complex multiple raster algebra operations.

• Default Image Processing Job Flow
  
• Multiple Raster Image Processing Job Flow
  

The following input parameters can be supplied to the hadoop command:

hadoop jar /opt/oracle/oracle-spatial-graph/spatial/raster/jlib/hadoop-imageprocessor.jar 
  -config  <MOSAIC_CONFIG_PATH>
  -gdal  <GDAL_LIBRARIES_PATH>
  -gdalData  <GDAL_DATA_PATH>
  [-catalog  <IMAGE_CATALOG_PATH>]
  [-usrlib  <USER_PROCESS_JAR_PATH>]
  [-thumbnail  <THUMBNAIL_PATH>]
  [-nativepath  <USER_NATIVE_LIBRARIES_PATH>]
  [-params  <USER_PARAMETERS>]
  [-file  <SINGLE_RASTER_PATH>]

Where:

• MOSAIC_CONFIG_PATH is the path to the mosaic configuration xml, that defines the features of the output.
• GDAL_LIBRARIES_PATH is the path where GDAL libraries are located.
• GDAL_DATA_PATH is the path where the GDAL data folder is located. This path must be accessible via NFS to all nodes in the cluster.
• IMAGE_CATALOG_PATH is the path to the catalog xml that lists the HDFS image(s) to be processed. This is optional because you can also specify a single raster to process using –file flag.
• USER_PROCESS_JAR_PATH is an optional user-defined jar file or comma-separated list of jar files, each of which contains additional processing classes to be applied to the source images.
• THUMBNAIL_PATH is an optional flag to activate the thumbnail creation of the loaded image(s). This path must be accessible via NFS to all nodes in the cluster and is valid only for an HDFS output.
• USER_NATIVE_LIBRARIES_PATH is an optional comma-separated list of additional native libraries to use in the analysis. It can also be a directory containing all the native libraries to load in the application.
• USER_PARAMETERS is an optional key/value list used to define input data for user processing classes. Use a semicolon to separate parameters. For example: azimuth=315;altitude=45
• SINGLE_RASTER_PATH is an optional path to the .ohif file that will be processed by the job. If this is set, you do not need to set a catalog.

For example, the following command will process all the files listed in the catalog file input.xml file using the mosaic output definition set in testFS.xml file.

hadoop jar /opt/oracle/oracle-spatial-graph/spatial/raster/jlib/hadoop-imageprocessor.jar -catalog /opt/shareddir/spatial/demo/imageserver/images/input.xml -config /opt/shareddir/spatial/demo/imageserver/images/testFS.xml -thumbnail /opt/shareddir/spatial/processtest –gdal /opt/oracle/oracle-spatial-graph/spatial/raster/gdal/lib –gdalData /opt/shareddir/data

By default, the Mappers and Reducers are configured to get 2 GB of JVM, but users can override this settings or any other job configuration properties by adding an imagejob.prop properties file in the same folder location from where the command is being executed.

For GDAL to work properly, the libraries must be available using $LD_LIBRARY_PATH. Make sure that the shared libraries path is set properly in your shell window before executing a job. For example:

export LD_LIBRARY_PATH=$ALLACCESSDIR/gdal/native

• Catalog XML Structure
  
• Mosaic Definition XML Structure
  

The first step of the job is to filter the tiles that would fit into the output. As a start, the location files that hold tile metadata are sent to theInputFormat.

By extracting the pixelType, the filter decides whether the related source image is valid for processing or not. Based on the user definition made in the catalog xml, one of the following happens:

• If the image is valid for processing, then the SRID is evaluated next

• If it is different from the user definition, then the MBR coordinates of every tile are converted into the user SRID and evaluated.

This way, every tile is evaluated for intersection with the output definition.

• For a mosaic processing request, only the intersecting tiles are selected, and a split is created for each one of them.

• For a single raster processing request, all the tiles are selected, and a split is created for each one of them.

• For a complex multiple raster algebra processing request, all the tiles are selected if the MBR and pixel size is the same. Depending on the number of rasters selected and the blocksize, a specific area of every tile´s raster (which does not always include the complete original raster tile) is included in a single parent split.

A mapper processes each split in the node where it is stored. (For complex multiple raster algebra operations, data locality may be lost, because a split contains data from several rasters.) The mapper executes the sequence of map algebra operations and processing classes defined by the user, and then the mosaic process is executed if requested. A single reducer puts together the result of the mappers and, for user-specified reducing processing classes, sets the output data set to these classes for analysis or process. Finally, the process stores the image into FS or HDFS upon user request. If the user requested to store the output into HDFS, then the ImageLoader job is invoked to store the image as an .ohif file.

By default, the mappers and reducers are configured to get 1 GB of JVM, but you can override this settings or any other job configuration properties by adding an imagejob.prop properties file in the same folder location from where the command is being executed.

The processing classes specified in the catalog XML must follow a set of rules to be correctly processed by the job. All the processing classes in the mapping phase must implement the ImageProcessorInterface interface. For the reducer phase, they must implement the ImageProcessorReduceInterface interface.

When implementing a processing class, you may manipulate the raster using its object representation ImageBandWritable. An example of an processing class is provided with the framework to calculate the slope on DEMs. You can create mapping operations, for example, to transforms the pixel values to another value by a function. The ImageBandWritable instance defines the content of a tile, such as resolution, size, and pixels. These values must be reflected in the properties that create the definition of the tile. The integrity of the mosaic output depends on the correct manipulation of these properties.

The ImageBandWritable instance defines the content of a tile, such as resolution, size, and pixels. These values must be reflected in the properties that create the definition of the tile. The integrity of the output depends on the correct manipulation of these properties.

• Location of the Classes and Jar Files
  

You can process local map algebra operations on the input rasters, where pixels are altered depending on the operation. The order of operations in the configuration XML determines the order in which the operations are processed. After all the map algebra operations are processed, the processing classes are run, and finally the mosaic operation is performed.

The following map algebra operations can be added in the <operations> element in the mosaic configuration XML, with the operation name serving as an element name. (The data types for which each operation is supported are listed in parentheses.)

• localnot: Gets the negation of every pixel, inverts the bit pattern. If the result is a negative value and the data type is unsigned, then the NODATA value is set. If the raster does not have a specified NODATA value, then the original pixel is set. (Byte, Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits)

• locallog: Returns the natural logarithm (base e) of a pixel. If the result is NaN, then original pixel value is set; if the result is Infinite, then the NODATA value is set. If the raster does not have a specified NODATA value, then the original pixel is set. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• locallog10: Returns the base 10 logarithm of a pixel. If the result is NaN, then the original pixel value is set; if the result is Infinite, then the NODATA value is set. If the raster does not have a specified NODATA value, then the original pixel is set. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localadd: Adds the specified value as argument to the pixel .Example: <localadd arg="5"/>. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localdivide: Divides the value of each pixel by the specified value set as argument. Example: <localdivide arg="5"/>. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localif: Modifies the value of each pixel based on the condition and value specified as argument. Valid operators: = , <, >, >=, < !=. Example: <localif operator="<" operand="3" newvalue="6"/>, which modifies all the pixels whose value is less than 3, setting the new value to 6. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localmultiply: Multiplies the value of each pixel times the value specified as argument. Example: <localmultiply arg="5"/>. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localpow: Raises the value of each pixel to the power of the value specified as argument. Example: <localpow arg="5"/>. If the result is infinite, the NODATA value is set to this pixel. If the raster does not have a specified NODATA value, then the original pixel is set. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localsqrt: Returns the correctly rounded positive square root of every pixel. If the result is infinite or NaN, the NODATA value is set to this pixel. If the raster does not have a specified NODATA value, then the original pixel is set. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localsubstract: Subtracts the value specified as argument to every pixel value. Example: <localsubstract arg="5"/>. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localacos: Calculates the arc cosine of a pixel. If the result is NaN, the NODATA value is set to this pixel. If the raster does not have a specified NODATA value, then the original pixel is set. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localasin: Calculates the arc sine of a pixel. If the result is NaN, the NODATA value is set to this pixel. If the raster does not have a specified NODATA value, then the original pixel is set. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localatan: Calculates the arc tangent of a pixel. If the result is NaN, the NODATA value is set to this pixel. If the raster does not have a specified NODATA value, then the original pixel is set. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localcos: Calculates the cosine of a pixel. If the result is NaN, the NODATA value is set to this pixel. If the raster does not have a specified NODATA value, then the original pixel is set. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localcosh: Calculates the hyperbolic cosine of a pixel. If the result is NaN, the NODATA value is set to this pixel. If the raster does not have a specified NODATA value, then the original pixel is set. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localsin: Calculates the sine of a pixel. If the result is NaN, the NODATA value is set to this pixel. If the raster does not have a specified NODATA value, then the original pixel is set. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localtan: Calculates the tangent of a pixel. The pixel is not modified if the cosine of this pixel is 0. If the result is NaN, the NODATA value is set to this pixel. If the raster does not have a specified NODATA value, then the original pixel is set. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localsinh: Calculates the arc hyperbolic sine of a pixel. If the result is NaN, the NODATA value is set to this pixel. If the raster does not have a specified NODATA value, then the original pixel is set. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localtanh: Calculates the hyperbolic tangent of a pixel. If the result is NaN, the NODATA value is set to this pixel. If the raster does not have a specified NODATA value, then the original pixel is set. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localdefined: Maps an integer typed pixel to 1 if the cell value is not NODATA; otherwise, 0. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits, Float 32 bits)

• localundefined: Maps an integer typed Raster to 0 if the cell value is not NODATA; otherwise, 1. (Unsigned int 16 bits, Unsigned int 32 bits, Int 16 bits, Int 32 bits)

• localabs: Returns the absolute value of signed pixel. If the result is Infinite, the NODATA value is set to this pixel. If the raster does not have a specified NODATA value, then the original pixel is set. (Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localnegate: Multiplies by -1 the value of each pixel. (Int 16 bits, Int 32 bits, Float 32 bits, Float 64 bits)

• localceil: Returns the smallest value that is greater than or equal to the pixel value and is equal to a mathematical integer. If the result is Infinite, the NODATA value is set to this pixel. If the raster does not have a specified NODATA value, then the original pixel is set. (Float 32 bits, Float 64 bits)

• localfloor: Returns the smallest value that is less than or equal to the pixel value and is equal to a mathematical integer. If the result is Infinite, the NODATA value is set to this pixel. If the raster does not have a specified NODATA value, then the original pixel is set. (Float 32 bits, Float 64 bits)

• localround: Returns the closest integer value to every pixel. (Float 32 bits, Float 64 bits)

You can process raster algebra operations that involve more than one raster, where pixels are altered depending on the operation and taking in consideration the pixels from all the involved rasters in the same cell.

Only one operation can be processed at a time and it is defined in the configuration XML using the <multipleops> element. Its value is the operation to process.

There are two types of operations:

• Basic Multiple Raster Algebra Operations are executed in the reduce phase right before the Reduce User Processing classes.

• Complex Multiple Raster Algebra Operations are processed in the mapping phase.

• Basic Multiple Raster Algebra Operations
  
• Complex Multiple Raster Algebra Operations
  

Pyramids are subobjects of a raster object that represent the raster image or raster data at differing sizes and degrees of resolution.

The size is usually related to the amount of time that an application needs to retrieve and display an image, particularly over the web. That is, the smaller the image size, the faster it can be displayed; and as long as detailed resolution is not needed (for example, if the user has "zoomed out" considerably), the display quality for the smaller image is adequate.

Pyramid levels represent reduced or increased resolution images that require less or more storage space, respectively. (Big Data Spatial and Graph supports only reduced resolution pyramids.) A pyramid level of 0 indicates the original raster data; that is, there is no reduction in the image resolution and no change in the storage space required. Values greater than 0 (zero) indicate increasingly reduced levels of image resolution and reduced storage space requirements.

A single raster is processed for each pyramid request, and the following parameters apply:

• Pyramid level (required): the maximum reduction level; that is, the number of pyramid levels to create at a reduced size than the original object. For example, redLevel=”6” causes pyramid levels to be created for levels 0 through 5.

  The dimension sizes at each lower level are: r(n) = r(n - 1)/2 and c(n) = c(n - 1)/2 where:

  r(n) and c(n) are the row and column sizes for a pyramid at level n

  The smaller of the row and column dimension sizes of the top-level overview is between 64 and 128 (maximum reduced-resolution level): (int)(log2(a/64)) where a is the smaller of the original row or column dimension size.

  If an rLevel value greater than the maximum reduced-resolution level is specified, the rLevel value is set to the maximum reduced-resolution level.

• Resampling algorithm: the resampling method to use.

  Must be one of the following: NEAREST_NEIGHBOR, BILINEAR, AVERAGE4, AVERAGE16. (BILINEAR and AVERAGE4 have the same effect.) If no resampling algorithm is specified, BILINEAR is used by default.

Pyramids can be created while loading multiple rasters or processing a single raster:

• While loading the rasters in HDFS, by adding the -pyramid parameter to the loader command line call or by using the API loader.addPyramid()

• For processing a single raster, by adding the operation in the user request XML or by using the API processor.addPyramid()

When you specify an HDFS directory in the configuration XML, the output generated is an .ohif file as in the case of an ImageLoader job,

When the user specifies a FS directory in the configuration XML, the output generated is an image with the filename and type specified and is stored into regular FileSystem.

In both the scenarios, the output must comply with the specifications set in the configuration XML. The job execution logs can be accessed using the command yarn logs -applicationId <applicationId>.

The first step in using the raster processing for Spark Java API is to have the images in HDFS, followed by having the images separated into smart tiles. This allows the processor to work on each tile independently. The Spark raster loader lets you import a single image or a collection of them into HDFS in parallel, which decreases the load time. Each block contains data for all the raster bands, so that if further processing is required on specific pixels, the information can be processed on a single node.

The basic workflow for the Spark raster loader is as follows.

1. GDAL is used to import the rasters, tiling them according to block size and then storing each tile as an HDFS block.

2. The set of rasters to be loaded is read into a SpatialRasterJavaRDD, which is an extension of JavaRDD. This RDD is a collection of ImagePieceWritable objects that represent the information of the tiles to create per raster, based on the number of bands, pixel size, HDFS block size, and raster resolution. This is accomplished by using the custom input format used in the spatial Hadoop loader.

3. The raster information for each tile is loaded. This load is performed by an executor for each tile, so reading is performed parallel. Each tile includes a certain number of overlapping bytes (user input), so that the tiles cover area from the adjacent tiles. There are “n” number of Spark executors, depending on the number of tiles, image resolution, and block size.

4. The RDD is grouped by key, so that all the tiles that correspond to the same raster are part of the same record. This RDD is saved as OHIF using the OhifOutputFormat, which puts together all the information loaded by the executors and stores the images into a special .ohif format, which contains the resolution, bands, offsets, and image data. In this way, the file offset containing each tile and the node location is known. A special reading process is required to read the image back and is included in the Spark SQL raster processor.

Each tile contains information for every band. This is helpful when there is a need to process only a few tiles; then, only the corresponding blocks are loaded.

The loader can be configured by setting parameters on the command line or by using the Spark API.

• Input Parameters to the Spark Raster Loader
  
• Expected Output of the Spark Raster Loader
  

Once the images are loaded into HDFS, they can be processed using Spark SQL Raster Processor. You specify the expected raster output features using the Mosaic Definition XML Structure or the Spark API, and the mosaic UDF filters the tiles to fit into that output and processes them. Raster algebra operations are also available in UDF.

A custom InputFormat, which is also used in the Hadoop raster processing framework, loads specific blocks of data, based on the input (mosaic description or a single raster) using raster SRID and coordinates, and selects only the bands and pixels that fit into the final output before accepting processing operations:

• For a mosaic processing request, only the intersecting tiles are selected, and a split is created for each one of them.

• For a single raster processing request, all the tiles are selected, and a split is created for each one of them.

The Spark SQL Raster Processor allows you to filter the OHIF tiles based on input catalog or raster into a Dataframe, with every row representing a tile, and to use Spatial UDF Spark functions to process them.

A simplified pseudocode representation of Spark SQL raster processing is:

sqlContext.udf().register("localop", new LocalOperationsFunction(),DataTypes.createStructType(SpatialRasterJavaRDD.createSimpleTileStructField(dataTypeOfTileToProcess)));
tileRows.registerTempTable("tiles");
String query = "SELECT localop(tileInfo, userRequest,  \"localnot\"), userRequest FROM tiles";
DataFrame processedTiles = sqlContext.sql(query);

The basic workflow of the Spark SQL raster processor is as follows.

1. The rasters to process are first loaded in tiles metadata as RDD. These tiles may be filtered if the user set a configuration for mosaic operation. The RDD is later converted to a Spark DataFrame (or Dataset<Row> for Spark 2.2) of two complex rows: the first row is tileInfo, which has all the metadata for the tiles, and the second row is the userRequest, which has the user input configuration listing the expected features of the raster output.

2. Once the DataFrame or Dataset<Row> is created, the driver must register the “localop” UDF, and also register the DataFrame or Dataset<Row> as a table before executing a query to process. The mosaic UDF can only be executed if the user configured all the required parameters correctly. If no XML is used and the configuration is set using the API, then by default a mosaic operation configuration is expected unless the setExecuteMosaic(false) method is set.

3. The mosaic operation selects from every tile only the pixels that fit into the output, and makes the necessary resolution changes to add them in the mosaic output.

4. Once the query is executed, an executor loads the data corresponding tile, conserving data locality, and the specified local raster algebra operation is executed.

5. The row in the DataFrame or Dataset<Row> is updated with the new pixel data and returned to the driver for further processing if required.

6. Once the processing is done, the DataFrame or Dataset<Row> is converted to a list of ImageBandWritable objects, which are the MapReduce representation of processed tiles. These are input to the ProcessedRasterCreator, where resulting bytes of local raster algebra and/or mosaic operations are put together, and a final raster is stored into HDFS or the regular file system depending on the user request.

Only images with same data type (pixel depth) as the user configuration input data type (pixel depth) are considered. Only the tiles that intersect with coordinates specified by the user for the mosaic output are included. For processing of a single raster, the filter includes all the tiles of the input rasters, because the processing will be executed on the complete images.

• Input Parameters to the Spark SQL Raster Processor
  
• Expected Output of the Spark SQL Raster Processor
  

You can use the Spark raster API to load and process rasters by creating the driver class.

Some example classes are provided under /opt/oracle/oracle-spatial-graph/spatial/raster/examples/java/src. The /opt/oracle/oracle-spatial-graph/spatial/raster/examples/java/cmd directory also contains scripts to execute these examples from command line.

After executing the scripts and validated the results, you can modify the Java source files to experiment on them and compile them using the provided script /opt/oracle/oracle-spatial-graph/spatial/raster/examples/java/build.xml. Ensure that there is write access on the /opt/oracle/oracle-spatial-graph/spatial/raster/jlib directory.

For GDAL to work properly, the libraries must be available using $LD_LIBRARY_PATH. Make sure that the shared libraries path is set properly in your shell window before executing a job. For example:

export LD_LIBRARY_PATH=$ALLACCESSDIR/gdal/native

• Using the Spark Raster Loader API
  
• Configuring for Using the Spark SQL Processor API
  
• Creating the DataFrame or Dataset<Row>
  
• Using the Spark SQL UDF for Raster Algebra Operations
  

Oracle Big Data Spatial Vector Analysis provides classes for both the old and new (context objects) Hadoop APIs. In general, classes in the mapred package are used with the old API, while classes in the mapreduce package are used with the new API

The examples in this guide use the old Hadoop API; however, all the old Hadoop Vector API classes have equivalent classes in the new API. For example, the old class oracle.spatial.hadoop.vector.mapred.job.SpatialIndexing has the equivalent new class named oracle.spatial.hadoop.vector.mapreduce.job.SpatialIndexing. In general, and unless stated otherwise, only the change from mapred to mapreduce is needed to use the new Hadoop API Vector classes.

Classes such as oracle.spatial.hadoop.vector.RecordInfo, which are not in the mapred or mapreduce package, are compatible with both Hadoop APIs.

A spatial index is in the form of a key/value pair and generated as a Hadoop MapFile. Each MapFile entry contains a spatial index for one split of the original data. The key and value pair contains the following information:

• Key: a split identifier in the form: path + start offset + length.

• Value: a spatial index structure containing the actual indexed records.

The following figure depicts a spatial index in relation to the user data. The records are represented as r1, r2, and so on. The records are grouped into splits (Split 1, Split 2, Split 3, Split n). Each split has a Key-Value pair where the key identifies the split and the value identifies an Rtree index on the records in that split.

Description of the illustration spatial_index_rep.png

• Spatial Indexing Class Structure
  
• Configuration for Creating a Spatial Index
  
• Spatial Index Metadata
  
• Input Formats for a Spatial Index
  
• Support for GeoJSON and Shapefile Formats
  
• Removing a Spatial Index
  

MVSuggest can be used at the time of spatial indexing to get an approximate location for records that do not have geometry but have some text field. This text field can be used to determine the record location. The geometry returned by MVSuggest is used to include the record in the spatial index.

Because it is important to know the field containing the search text for every record, the RecordInfoProvider implementation must also implement LocalizableRecordInfoProvider. Alternatively, the configuration parameter oracle.spatial.recordInfo.locationField can be set with the name of the field containing the search text. For more information, see the Javadoc for LocalizableRecordInfoProvider.

A standalone version of MVSuggest is shipped with the Vector API and it can be used in some jobs that accept the MVSConfig as an input parameter.

The following job drivers can work with MVSuggest and all of them have the setMVSConfig() method which accepts an instance of MVSConfig:

• oracle.spatial.hadoop.vector.mapred.job.SpatialIndexing: has the option of using MVSuggest to get approximate spatial location for records which do not contain geometry.

• oracle.spatial.hadoop.vector.mapred.job.Categorization: MVSuggest can be used to assign a record to a specific feature in a layer, for example, the feature California in the USA states layer.

• oracle.spatial.hadoop.vector.mapred.job.SuggestService: A simple job that generates a file containing a search text and its match per input record.

The MVSuggest configuration is passed to a job using the MVSConfig or the LocalMVSConfig classes. The basic MVSuggest properties are:

• serviceLocation: It is the minimum property required in order to use MVSuggest. It contains the path or URL where the MVSuggest directory is located or in the case of a URL, where the MVSuggest service is deployed.

• serviceInterfaceType: the type of MVSuggest implementation used. It can be LOCAL(default) for a standalone version and WEB for the web service version.

• matchLayers: an array of layer names used to perform the searches.

When using the standalone version of MVSuggest, you must specify an MVSuggest directory or repository as the serviceLocation. An MVSuggest directory must have the following structure:

mvsuggest_config.json
repository folder
   one or more layer template files in .json format
   optionally, a _config_ directory
   optionally, a _geonames_ directory

The examples folder comes with many layer template files and a _config_ directory with the configuration for each template.

It is possible to set the repository folder (the one that contains the templates) as the mvsLocation instead of the whole MVSuggest directory. In order to do that, the class LocalMVSConfig can be used instead of MVSConfig and the repositoryLocation property must be set to true as shown in the following example:

LocalMVSConfig lmvsConf = new LocalMVSConfig();
lmvsConf.setServiceLocation(“file:///home/user/mvs_dir/repository/”);
lmvsConf.setRepositoryLocation(true);
lmvsConf.setPersistentServiceLocation(“/user/hdfs/hdfs_mvs_dir”);
spatialIndexingJob.setMvsConfig(lmvsConf);

The preceding example sets a repository folder as the MVS service location. setRepositoryLocation is set to true to indicate that the service location is a repository instead of the whole MVSuggest directory. When the job runs, a whole MVSuggest directory will be created using the given repository location; the repository will be indexed and will be placed in a temporary folder while the job finishes. The previously indexed MVSuggest directory can be persisted so it can be used later. The preceding example saves the generated MVSuggest directory in the HDFS path /user/hdfs/hdfs_mvs_dir. Use the MVSDirectory if the MVSuggest directory already exists.

Once the spatial index has been generated, it can be used to spatially filter the data. The filtering is performed before the data reaches the mapper and while it is being read. The following sample code example demonstrates how the SpatialFilterInputFormat is used to spatially filter the data.

//set input path and format
 
FileInputFormat.setInputPaths(conf, new Path("/user/data/"));
conf.setInputFormat(SpatialFilterInputFormat.class);
 
//set internal input format
 
SpatialFilterInputFormat.setInternalInputFormatClass(conf, TextInputFormat.class);
if( spatialIndexPath != null )  
{
 
     //set the path to the spatial index and put it in the distributed cache
 
     boolean useDistributedCache = true;
     SpatialFilterInputFormat.setSpatialIndexPath(conf, spatialIndexPath, useDistributedCache);
} 
else 
{
     //as no spatial index is used a RecordInfoProvider is needed
 
     SpatialFilterInputFormat.setRecordInfoProviderClass(conf, TwitterLogRecordInfoProvider.class);
}
 
//set spatial operation used to filter the records
 
SpatialOperationConfig spatialOpConf = new SpatialOperationConfig();
spatialOpConf.setOperation(SpatialOperation.IsInside);
spatialOpConf.setJsonQueryWindow("{\"type\":\"Polygon\", \"coordinates\":[[-106.64595, 25.83997, -106.64595, 36.50061, -93.51001, 36.50061, -93.51001, 25.83997 , -106.64595, 25.83997]]}");
spatialOpConf.setSrid(8307);
spatialOpConf.setTolerance(0.5);
spatialOpConf.setGeodetic(true);

SpatialFilterInputFormat has to be set as the job's InputFormat. The InputFormat that actually reads the data must be set as the internal InputFormat. In this example, the internal InputFormat is TextInputFormat.

If a spatial index is specified, it is used for filtering. Otherwise, a RecordInfoProvider must be specified in order to get the records geometries, in which case the filtering is performed record by record.

As a final step, the spatial operation and query window to perform the spatial filter are set. It is recommended to use the same internal InputFormat implementation used when the spatial index was created or, at least, an implementation that uses the same criteria to generate the splits. For details see "Input Formats for a Spatial Index."

If a simple spatial filtering needs to be performed (that is, only retrieving records that interact with a query window), the built-in job driver oracle.spatial.hadoop.vector.mapred.job.SpatialFilter can be used instead. This job driver accepts indexed or non-indexed input and a SpatialOperationConfig to perform the filtering.

• Filtering Records
  
• Filtering Using the Input Format
  

The Vector Analysis API provides a way to classify the data into hierarchical entities. For example, in a given set of catalogs with a defined level of administrative boundaries such as continents, countries and states, it is possible to join a record of the user data to a record of each level of the hierarchy data set. The following example generates a summary count for each hierarchy level, containing the number of user records per continent, country, and state or province:

Categorization catJob = new Categorization();
//set a spatial index as the input
				 
catJob.setIndexName("indexExample");
 
//set the job's output
				 
catJob.setOutput("hierarchy_count");
				 
//set HierarchyInfo implementation which describes the world administrative boundaries hierarchy
			
catJob.setHierarchyInfoClass( WorldDynaAdminHierarchyInfo.class );
				 
//specify the paths of the hierarchy data
				 
Path[] hierarchyDataPaths = {
				new Path("file:///home/user/catalogs/world_continents.json"), 
				new Path("file:///home/user/catalogs/world_countries.json"), 
				new Path("file:///home/user/catalogs/world_states_provinces.json")};
catJob.setHierarchyDataPaths(hierarchyDataPaths);
				 
//set the path where the index for the previous hierarchy data will be generated
				 
catJob.setHierarchyIndexPath(new Path("/user/hierarchy_data_index/"));
				 
//setup the spatial operation which will be used to join records from the two datasets (spatial index and hierarchy data).
SpatialOperationConfig spatialOpConf = new SpatialOperationConfig();			 
spatialOpConf.setOperation(SpatialOperation.IsInside);
spatialOpConf.setSrid(8307);
spatialOpConf.setTolerance(0.5);
spatialOpConf.setGeodetic(true);
catJob.setSpatialOperationConfig(spatialOpConf);
				 
//add the previous setup to the job configuration
				 
catJob.configure(conf);
				 
//run the job 
RunningJob rj = JobClient.runJob(conf);

The preceding example uses the Categorization job driver. The configuration can be divided into the following categories:

• Input data: A previously generated spatial index (received as the job input).

• Output data: A folder that contains the summary counts for each hierarchy level.

• Hierarchy data configuration: This contains the following:

  • HierarchyInfo class: This is an implementation of HierarchyInfo class in charge of describing the current hierarchy data. It provides the number of hierarchy levels, level names, and the data contained at each level.

  • Hierarchy data paths: This is the path to each one of the hierarchy catalogs. These catalogs are read by the HierarchyInfo class.

  • Hierarchy index path: This is the path where the hierarchy data index is stored. Hierarchy data needs to be preprocessed to know the parent-child relationships between hierarchy levels. This information is processed once and saved at the hierarchy index, so it can be used later by the current job or even by any other jobs.

• Spatial operation configuration: This is the spatial operation to be performed between records of the user data and the hierarchy data in order to join both datasets. The parameters to set here are the Spatial Operation type (IsInside), SRID (8307), Tolerance (0.5 meters), and whether the geometries are Geodetic (true).

Internally, the Categorization.configure() method sets the mapper and reducer to be SpatialHierarchicalCountMapper and SpatialHierarchicalCountReducer, respectively. SpatialHierarchicalCountMapper's output key is a hierarchy entry identifier in the form hierarchy_level + hierarchy_entry_id. The mapper output value is a single count for each output key. The reducer sums up all the counts for each key.

If you want another type of output instead of counts, for example, a list of user records according to the hierarchy entry. In this case, the SpatialHierarchicalJoinMapper can be used. The SpatialHierarchicalJoinMapper output value is a RecordInfo instance, which can be gathered in a user-defined reducer to produce a different output. The following user-defined reducer generates a MapFile for each hierarchy level using the MultipleOutputs class. Each MapFile has the hierarchy entry ids as keys and ArrayWritable instances containing the matching records for each hierarchy entry as values. The following is an user-defined reducer that returns a list of records by hierarchy entry:

public class HierarchyJoinReducer extends MapReduceBase implements Reducer<Text, RecordInfo, Text, ArrayWritable> {
 
       private MultipleOutputs mos = null;
       private Text outKey = new Text();
       private ArrayWritable outValue = new ArrayWritable( RecordInfo.class );
 
       @Override
       public void configure(JobConf conf) 
       {
         super.configure(conf);
 
         //use MultipleOutputs to generate different outputs for each hierarchy level
    
         mos = new MultipleOutputs(conf);
        }
        @Override
        public void reduce(Text key, Iterator<RecordInfo> values,
                          OutputCollector<Text, RecordInfoArrayWritable> output, Reporter reporter)
                          throws IOException 
        {
 
          //Get the hierarchy level name and the hierarchy entry id from the key
 
          String[] keyComponents = HierarchyHelper.getMapRedOutputKeyComponents(key.toString());
          String hierarchyLevelName = keyComponents[0];
          String entryId = keyComponents[1];
          List<Writable> records = new LinkedList<Writable>();
 
          //load the values to memory to fill output ArrayWritable
      
          while(values.hasNext())
          {
            RecordInfo recordInfo = new RecordInfo( values.next() );
            records.add( recordInfo );		
          }
          if(!records.isEmpty())
          {
 
            //set the hierarchy entry id as key
 
            outKey.set(entryId);
 
            //list of records matching the hierarchy entry id
 
            outValue.set( records.toArray(new Writable[]{} ) );
 
            //get the named output for the given hierarchy level
 
            hierarchyLevelName = FileUtils.toValidMONamedOutput(hierarchyLevelName);
            OutputCollector<Text, ArrayWritable> mout = mos.getCollector(hierarchyLevelName, reporter);
 
           //Emit key and value
 
           mout.collect(outKey, outValue);
          }
}
 
        @Override
        public void close() throws IOException 
        {
          mos.close();
        }
}

The same reducer can be used in a job with the following configuration to generate a list of records according to the hierarchy levels:

JobConf conf = new JobConf(getConf());
 
//input path
 
FileInputFormat.setInputPaths(conf, new Path("/user/data_spatial_index/") );
 
//output path
 
FileOutputFormat.setOutputPath(conf, new Path("/user/records_per_hier_level/") );
 
//input format used to read the spatial index
 
conf.setInputFormat( SequenceFileInputFormat.class);
 
//output format: the real output format will be configured for each multiple output later
 
conf.setOutputFormat(NullOutputFormat.class);
 
//mapper
 
conf.setMapperClass( SpatialHierarchicalJoinMapper.class );
conf.setMapOutputKeyClass(Text.class);
conf.setMapOutputValueClass(RecordInfo.class);
 
//reducer
 
conf.setReducerClass( HierarchyJoinReducer.class );
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(ArrayWritable.class);
 
////////////////////////////////////////////////////////////////////
 
//hierarchy data setup
 
//set HierarchyInfo class implementation
 
conf.setClass(ConfigParams.HIERARCHY_INFO_CLASS, WorldAdminHierarchyInfo.class, HierarchyInfo.class);
 
//paths to hierarchical catalogs
 
Path[] hierarchyDataPaths = {
new Path("file:///home/user/catalogs/world_continents.json"), 
new Path("file:///home/user/catalogs/world_countries.json"), 
new Path("file:///home/user/catalogs/world_states_provinces.json")};
 
//path to hierarchy index
 
Path hierarchyDataIndexPath = new Path("/user/hierarchy_data_index/");
 
//instantiate the HierarchyInfo class to index the data if needed.
 
HierarchyInfo hierarchyInfo = new WorldAdminHierarchyInfo();
hierarchyInfo.initialize(conf);
 
//Create the hierarchy index if needed. If it already exists, it will only load the hierarchy index to the distributed cache
 
HierarchyHelper.setupHierarchyDataIndex(hierarchyDataPaths, hierarchyDataIndexPath, hierarchyInfo, conf);
 
///////////////////////////////////////////////////////////////////
 
//setup the multiple named outputs:
 
int levels = hierarchyInfo.getNumberOfLevels();
for(int i=1; i<=levels; i++)
{
    String levelName = hierarchyInfo.getLevelName(i);
 
    //the hierarchy level name is used as the named output
 
    String namedOutput = FileUtils.toValidMONamedOutput(levelName);
    MultipleOutputs.addNamedOutput(conf, namedOutput, MapFileOutputFormat.class, Text.class, ArrayWritable.class);
}
 
//finally, setup the spatial operation
 
SpatialOperationConfig spatialOpConf = new SpatialOperationConfig();			 
spatialOpConf.setOperation(SpatialOperation.IsInside);
spatialOpConf.setSrid(8307);
spatialOpConf.setTolerance(0.5);
spatialOpConf.setGeodetic(true);
spatialOpConf.store(conf);
 
//run job
 
JobClient.runJob(conf);

Supposing the output value should be an array of record ids instead of an array of RecordInfo instances, it would be enough to perform a couple of changes in the previously defined reducer.

The line where outValue is declared, in the previous example, changes to:

private ArrayWritable outValue = new ArrayWritable(Text.class);

The loop where the input values are retrieved, in the previous example, is changed. Therefore, the record ids are got instead of the whole records:

while(values.hasNext())
{
  records.add( new Text(values.next().getId()) );
}

While only the record id is needed the mapper emits the whole RecordInfo instance. Therefore, a better approach is to change the mappers output value. The mappers output value can be changed by extending AbstractSpatialJoinMapper. In the following example, the mapper emits only the record ids instead of the whole RecorInfo instance every time a record matches some of the hierarchy entries:

public class IdSpatialHierarchicalMapper extends AbstractSpatialHierarchicalMapper< Text > 
{
       Text outValue = new Text();
 
       @Override
       protected Text getOutValue(RecordInfo matchingRecordInfo) 
       {
 
         //the out value is the record's id
 
         outValue.set(matchingRecordInfo.getId());
         return outValue;
       }
}

• Changing the Hierarchy Level Range
  
• Controlling the Search Hierarchy
  
• Using MVSuggest to Classify the Data
  

The Vector API provides the class oracle.spatial.hadoop.vector.mapred.job.Binning to perform spatial binning over a spatial data set. The Binning class is a MapReduce job driver that takes an input data set (which can be spatially indexed or not), assigns each record to a bin, and generates a file containing all the bins (which contain one or more records and optionally aggregated values).

A binning job can be configured as follows:

1. Specify the data set to be binned and the way it will be read and interpreted (InputFormat and RecordInfoProvider), or, specify the name of an existing spatial index.

2. Set the output path.

3. Set the grid MBR, that is, the rectangular area to be binned.

4. Set the shape of the bins: RECTANGLE or HEXAGON.

5. Specify the bin (cell) size. For rectangles, specify the width and height. For hexagon-shaped cells, specify the hexagon width. Each hexagon is always drawn with only one of its vertices as the base.

6. Optionally, pass a list of numeric field names to be aggregated per bin.

The resulting output is a text file where each record is a bin (cell) in JSON format and contains the following information:

• id: the bin id

• geom: the bin geometry; always a polygon that is a rectangle or a hexagon

• count: the number of points contained in the bin

• aggregated fields: zero or more aggregated fields

The following example configures and runs a binning job:

//create job driver
Binning<LongWritable, Text> binJob = new Binning<LongWritable, Text>();
//setup input
binJob.setInput("/user/hdfs/input/part*");
binJob.setInputFormatClass(GeoJsonInputFormat.class);
binJob.setRecordInfoProviderClass(GeoJsonRecordInfoProvider.class);
//set binning output
binJob.setOutput("/user/hdfs/output/binning");
//create a binning configuration to produce rectangular cells
BinningConfig binConf = new BinningConfig();
binConf.setShape(BinShape.RECTANGLE);
//set the bin size
binConf.setCellHeight(0.2);
binConf.setCellWidth(0.2);
//specify the area to be binned
binConf.setGridMbr(new double[]{-50,10,50,40});
binJob.setBinConf(binConf);
//save configuration
binJob.configure(conf);
//run job
JobClient.runJob(conf);

The job driver class oracle.spatial.hadoop.mapred.KMeansClustering can be used to find spatial clusters in a data set. This class uses a distributed version of the K-means algorithm.

Required parameters:

• Path to the input data set, the InputFormat class used to read the input data set and the RecordInfoProvider used to extract the spatial information from records.

• Path where the results will be stored.

• Number of clusters to be found.

Optional parameters:

• Maximum number of iterations before the algorithm finishes.

• Criterion function used to determine when the clusters converge. It is given as an implementation of oracle.spatial.hadoop.vector.cluster.kmeans.CriterionFunction. The Vector API contains the following criterion function implementations: SquaredErrorCriterionFunction and EuclideanDistanceCriterionFunction.

• An implementation of oracle.spatial.hadoop.vector.cluster.kmeans.ClusterShapeGenerator, which is used to generate a geometry for each cluster. The default implementation is ConvexHullClusterShapeGenerator and generates a convex hull for each cluster. If no cluster geometry is needed, the DummyClusterShapeGenerator class can be used.

• The initial k cluster points as a sequence of x,y ordinates. For example: x1,y1,x2,y2,…xk,yk

The result is a file named clusters.json, which contains an array of clusters called features. Each cluster contains the following information:

• id: Cluster id

• memberCount: Number of elements in the cluster

• geom: Cluster geometry

The following example runs the KMeansClustering algorithm to find 5 clusters. By default, the SquredErrorCriterionFunction and ConvexHullClusterShapeGenerator are used , so you do not need yo set these classes explicitly. Also note that runIterations() is called to run the algorithm; internally, it launches one MapReduce per iteration. In this example, the number 20 is passed to runIterations() as the maximum number of iterations allowed.

//create the cluster job driver
KMeansClustering<LongWritable, Text> clusterJob = new KMeansClustering<LongWritable, Text>();
//set input properties:
//input dataset path
clusterJob.setInput("/user/hdfs/input/part*");
//InputFormat class
clusterJob.setInputFormatClass(GeoJsonInputFormat.class);
//RecordInfoProvider implementation
clusterJob.setRecordInfoProviderClass(GeoJsonRecordInfoProvider.class);
//specify where the results will be saved
clusterJob.setOutput("/user/hdfs/output/clusters");
//5 cluster will be found
clusterJob.setK(5);
//run the algorithm
success = clusterJob.runIterations(20, conf);

A record read by a MapReduce job from HDFS is represented in memory as a key-value pair using a Java type (typically) Writable subclass, such as LongWritable, Text, ArrayWritable or some user-defined type. For example, records read using TextInputFormat are represented in memory as LongWritable, Text key-value pairs.

RecordInfoProvider is the component that interprets these memory record representations and returns the data needed by the Vector Analysis API. Thus, the API is not tied to any specific format and memory representations.

The RecordInfoProvider interface has the following methods:

• void setCurrentRecord(K key, V value)

• String getId()

• JGeometry getGeometry()

• boolean getExtraFields(Map<String, String> extraFields)

There is always a RecordInfoProvider instance per InputFormat. The method setCurrentRecord() is called passing the current key-value pair retrieved from the RecordReader. The RecordInfoProvider is then used to get the current record id, geometry, and extra fields. None of these fields are required fields. Only those records with a geometry participates in the spatial operations. The Id is useful for differentiating records in operations such as categorization. The extra fields can be used to store any record information that can be represented as text and which is desired to be quickly accessed without reading the original record, or for operations where MVSuggest is used.

Typically, the information returned by RecordInfoProvider is used to populate RecordInfo instances. A RecordInfo can be thought as a light version of a record and contains the information returned by the RecordInfoProvider plus information to locate the original record in a file.

• Sample RecordInfoProvider Implementation
  
• LocalizableRecordInfoProvider
  

The HierarchyInfo interface is used to describe a hierarchical dataset. This implementation of HierarchyInfo is expected to provide the number, names, and the entries of the hierarchy levels of the hierarchy it describes.

The root hierarchy level is always the hierarchy level 1. The entries in this level do not have parent entries and this level is referred as the top hierarchy level. Children hierarchy levels will have higher level values. For example: the levels for the hierarchy conformed by continents, countries, and states are 1, 2 and 3 respectively. Entries in the continent layer do not have a parent, but have children entries in the countries layer. Entries at the bottom level, the states layer, do not have children.

A HierarchyInfo implementation is provided out of the box with the Vector Analysis API. The DynaAdminHierarchyInfo implementation can be used to read and describe the known hierarchy layers in GeoJSON format. A DynaAdminHierarchyInfo can be instantiated and configured or can be subclassed. The hierarchy layers to be contained are specified by calling the addLevel() method, which takes the following parameters:

• The hierarchy level number

• The hierarchy level name, which must match the file name (without extension) of the GeoJSON file that contains the data. For example, the hierarchy level name for the file world_continents.json must be world_continents, for world_countries.json it is world_countries, and so on.

• Children join field: This is a JSON property that is used to join entries of the current level with child entries in the lower level. If a null is passed, then the entry id is used.

• Parent join field: This is a JSON property used to join entries of the current level with parent entries in the upper level. This value is not used for the top most level without an upper level to join. If the value is set null for any other level greater than 1, an IsInside spatial operation is performed to join parent and child entries. In this scenario, it is supposed that an upper level geometry entry can contain lower level entries.

For example, let us assume a hierarchy containing the following levels from the specified layers: 1- world_continents, 2 - world_countries and 3 - world_states_provinces. A sample entry from each layer would look like the following:

world_continents:
 {"type":"Feature","_id":"NA","geometry": {"type":"MultiPolygon", "coordinates":[ x,y,x,y,x,y] }"properties":{"NAME":"NORTH AMERICA", "CONTINENT_LONG_LABEL":"North America"},"label_box":[-118.07998,32.21006,-86.58515,44.71352]}

world_countries: {"type":"Feature","_id":"iso_CAN","geometry":{"type":"MultiPolygon","coordinates":[x,y,x,y,x,y]},"properties":{"NAME":"CANADA","CONTINENT":"NA","ALT_REGION":"NA","COUNTRY CODE":"CAN"},"label_box":[-124.28092,49.90408,-94.44878,66.89287]}

world_states_provinces:
{"type":"Feature","_id":"6093943","geometry": {"type":"Polygon", "coordinates":[ x,y,x,y,x,y]},"properties":{"COUNTRY":"Canada", "ISO":"CAN", "STATE_NAME":"Ontario"},"label_box":[-91.84903,49.39557,-82.32462,54.98426]}

A DynaAdminHierarchyInfo can be configured to create a hierarchy with the above layers in the following way:

DynaAdminHierarchyInfo dahi = new DynaAdminHierarchyInfo();
 
dahi.addLevel(1, "world_continents", null /*_id is used by default to join with child entries*/, null /*not needed as there are not upper hierarchy levels*/);
 
dahi.addLevel(2, "world_countries", "properties.COUNTRY CODE"/*field used to join with child entries*/, "properties.CONTINENT" /*the value "NA" will be used to find Canada's parent which is North America and which _id field value is also "NA" */);
 
dahi.addLevel(3, "world_states_provinces", null /*not needed as not child entries are expected*/, "properties.ISO"/*field used to join with parent entries. For Ontario, it is the same value than the field properties.COUNTRY CODE specified for Canada*/);
 
//save the previous configuration to the job configuration
 
dahi.initialize(conf);

A similar configuration can be used to create hierarchies from different layers, such as countries, states and counties, or any other layers with a similar JSON format.

Alternatively, to avoid configuring a hierarchy every time a job is executed, the hierarchy configuration can be enclosed in a DynaAdminHierarchyInfo subclass as in the following example:

public class WorldDynaAdminHierarchyInfo extends DynaAdminHierarchyInfo \
 
{
       public WorldDynaAdminHierarchyInfo() 
 
       {
              super();
              addLevel(1, "world_continents", null, null);
              addLevel(2, "world_countries", "properties.COUNTRY CODE", "properties.CONTINENT");
              addLevel(3, "world_states_provinces", null, "properties.ISO");
       }
 
}

• Sample HierarchyInfo Implementation
  

The Spatial Hadoop Vector Analysis only contains a small subset of the functionality provided by the Spatial Java API, which can also be used in the MapReduce jobs. This section provides some simple examples of how JGeometry can be used in Hadoop for spatial processing. The following example contains a simple mapper that performs the IsInside test between a dataset and a query geometry using the JGeometry class.

In this example, the query geometry ordinates, srid, geodetic value and tolerance used in the spatial operation are retrieved from the job configuration in the configure method. The query geometry, which is a polygon, is preprocessed to quickly perform the IsInside operation.

The map method is where the spatial operation is executed. Each input record value is tested against the query geometry and the id is returned, when the test succeeds.

public class IsInsideMapper extends MapReduceBase implements Mapper<LongWritable, Text, NullWritable, Text>
{
       private JGeometry queryGeom = null;
       private int srid = 0;
       private double tolerance = 0.0;
       private boolean geodetic = false;
       private Text outputValue = new Text();
       private double[] locationPoint = new double[2];
	
	
       @Override
       public void configure(JobConf conf) 
       {
           super.configure(conf);
           srid = conf.getInt("srid", 8307);
           tolerance = conf.getDouble("tolerance", 0.0);
           geodetic = conf.getBoolean("geodetic", true);
 
    //The ordinates are represented as a string of comma separated double values
           
            String[] ordsStr = conf.get("ordinates").split(",");
            double[] ordinates = new double[ordsStr.length];
            for(int i=0; i<ordsStr.length; i++)
            {
              ordinates[i] = Double.parseDouble(ordsStr[i]);
             }
 
    //create the query geometry as two-dimensional polygon and the given srid
 
             queryGeom = JGeometry.createLinearPolygon(ordinates, 2, srid);
 
    //preprocess the query geometry to make the IsInside operation run faster
 
              try 
              {
                queryGeom.preprocess(tolerance, geodetic, EnumSet.of(FastOp.ISINSIDE));
               } 
               catch (Exception e) 
               {
                 e.printStackTrace();
                }
		
          }
	
          @Override
          public void map(LongWritable key, Text value,
                      OutputCollector<NullWritable, Text> output, Reporter reporter)
                      throws IOException 
          {
 
     //the input value is a comma separated values text with the following columns: id, x-ordinate, y-ordinate
 
           String[] tokens = value.toString().split(",");
     
     //create a geometry representation of the record's location
 
            locationPoint[0] = Double.parseDouble(tokens[1]);//x ordinate
            locationPoint[1] = Double.parseDouble(tokens[2]);//y ordinate
            JGeometry location = JGeometry.createPoint(locationPoint, 2, srid);
 
     //perform spatial test
 
            try 
            {
              if( location.isInside(queryGeom, tolerance, geodetic)){
 
               //emit the record's id
 
               outputValue.set( tokens[0] );
               output.collect(NullWritable.get(), outputValue);
             }
     } 
             catch (Exception e) 
             {
                e.printStackTrace();
              }
}
}

A similar approach can be used to perform a spatial operation on the geometry itself. For example, by creating a buffer. The following example uses the same text value format and creates a buffer around each record location. The mapper output key and value are the record id and the generated buffer, which is represented as a JGeometryWritable. The JGeometryWritable is a Writable implementation contained in the Vector Analysis API that holds a JGeometry instance.

public class BufferMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, JGeometryWritable> 
{
       private int srid = 0;
       private double bufferWidth = 0.0;
       private Text outputKey = new Text();
       private JGeometryWritable outputValue = new JGeometryWritable();
       private double[] locationPoint = new double[2];
 
       @Override
       public void configure(JobConf conf)
       {
              super.configure(conf);
              srid = conf.getInt("srid", 8307);
 
              //get the buffer width 
 
              bufferWidth = conf.getDouble("bufferWidth", 0.0);
        }
 
        @Override
        public void map(LongWritable key, Text value,
               OutputCollector<Text, JGeometryWritable> output, Reporter reporter)
               throws IOException 
        {
 
               //the input value is a comma separated record with the following columns: id, longitude, latitude
 
               String[] tokens = value.toString().split(",");
 
               //create a geometry representation of the record's location
 
               locationPoint[0] = Double.parseDouble(tokens[1]);
               locationPoint[1] = Double.parseDouble(tokens[2]);
               JGeometry location = JGeometry.createPoint(locationPoint, 2, srid);
 
               try 
               {
 
                  //create the location's buffer
 
                  JGeometry buffer = location.buffer(bufferWidth);
 
                  //emit the record's id and the generated buffer
 
                  outputKey.set( tokens[0] );
                  outputValue.setGeometry( buffer );
                  output.collect(outputKey, outputValue);
                }
 
                catch (Exception e)
                {
                   e.printStackTrace();
                 }
        }
}

In addition to file-based data sources (that is, a file or a set of files from a local or a distributed file system), other types of data sources can be used as the input data for a Vector API job.

Data sources are referenced as input data sets in the Vector API. All the input data sets implement the interface oracle.spatial.hadoop.vector.data.AbstractInputDataSet. Input data set properties can be set directly for a Vector job using the methods setInputFormatClass(), setRecordInfoProviderClass(), and setSpatialConfig(). More information can be set, depending the type of input data set. For example, setInput() can specify the input string for a file data source, or setIndexName() can be used for a spatial index. The job determines the input data type source based on the properties that are set.

Input data set information can also be set directly for a Vector API job using the job’s method setInputDataSet(). With this method, the input data source information is encapsulated, you have more control, and it is easier to identify the type of data source that is being used.

The Vector API provides the following implementations of AsbtractInputDataSet:

• SimpleInputDataSet: Contains the minimum information required by the Vector API for an input data set. Typically, this type of input data set should be used for non-file based input data sets, such as Apache Hbase, an Oracle database, or any other non-file-based data source.

• FileInputDataSet: Encapsulates file-based input data sets from local or distributed file systems. It provides properties for setting the input path as an array of Path instances or as a string that can be a regular expression for selecting paths.

• SpatialIndexInputDataSet: A subclass of FileInputDataSet optimized for working with spatial indexes generated by the Vector API. It is sufficient to specify the index name for this type of input data set.

• NoSQLInputDataSet: Specifies Oracle NoSQL data sources. It should be used in conjunction with Vector NoSQL API. If the NoSQL KVInputFormat or TableInputFormat classes need to be used, use SimpleInputFormat instead.

• MultiInputDataSet: Input data set that encapsulates two or more input data sets.

//file input data set
FileInputDataSet fileDataSet = new FileInputDataSet();
fileDataSet.setInputFormatClass(GeoJsonInputFormat.class);
fileDataSet.setRecordInfoProviderClass(GeoJsonRecordInfoProvider.class);
fileDataSet.setInputString("/user/myUser/geojson/*.json");
		
//spatial index input data set
SpatialIndexInputDataSet indexDataSet = new SpatialIndexInputDataSet();
indexDataSet.setIndexName("myIndex");
		
//create multi input data set
MultiInputDataSet multiDataSet = new MultiInputDataSet();
		
//add the previously defined input data sets
multiDataSet.addInputDataSet(fileDataSet);
multiDataSet.addInputDataSet(indexDataSet);
		
Binning binningJob = new Binning();
//set multiple input data sets to the job
binningJob.setInputDataSet(multiDataSet);

//create NoSQL configuration
NoSQLConfiguration nsqlConf = new NoSQLConfiguration();
// set connection data
nsqlConf.setKvStoreName("mystore");
nsqlConf.setKvStoreHosts(new String[] { "myserver:5000" });
nsqlConf.setParentKey(Key.createKey("tweets"));
// set NoSQL entries to be included in the Hadoop records
// the entries with the following minor keys will be set as the
// RecordInfo's extra fields
nsqlConf.addTargetEntries(new String[] { "friendsCount", "followersCount" });
// add an entry processor to map the spatial entry to a RecordInfo's
// geometry
nsqlConf.addTargetEntry("geometry", NoSQLJGeometryEntryProcessor.class);
//create and set the NoSQL input data set
NoSQLInputDataSet nsqlDataSet = new NoSQLInputDataSet();
//set noSQL configuration
nsqlDataSet.setNoSQLConfig(nsqlConf);
//set spatial configuration
SpatialConfig spatialConf = new SpatialConfig();
spatialConf.setSrid(8307);
nsqlDataSet.setSpatialConfig(spatialConf);

//create job
Job job = Job.getInstance(getConf());
job.setJobName(getClass().getName());
job.setJarByClass(getClass());

//Setup hbase parameters
Scan scan = new Scan();
scan.setCaching(500);
scan.setCacheBlocks(false);
scan.addColumn(Bytes.toBytes("location_data"), Bytes.toBytes("geometry"));
scan.addColumn(Bytes.toBytes("other_data"), Bytes.toBytes("followers_count"));
scan.addColumn(Bytes.toBytes("other_data"), Bytes.toBytes("user_id"));

//initialize job configuration with hbase parameters
TableMapReduceUtil.initTableMapperJob(
		"tweets_table",
		scan,
		null,
		null,
		null,
		job);
//create binning job
Binning<ImmutableBytesWritable, Result> binningJob = new Binning<ImmutableBytesWritable, Result>();
//setup the input data set 
SimpleInputDataSet inputDataSet = new SimpleInputDataSet();
//use HBase's TableInputFormat
inputDataSet.setInputFormatClass(TableInputFormat.class);
//Set a RecordInfoProvider which can extract information from HBase TableInputFormat's returned key and values
inputDataSet.setRecordInfoProviderClass(HBaseRecordInfoProvider.class);
//set spatial configuration
SpatialConfig spatialConf = new SpatialConfig();
spatialConf.setSrid(8307);
inputDataSet.setSpatialConfig(spatialConf);
binningJob.setInputDataSet(inputDataSet);

//job output
binningJob.setOutput("hbase_example_output");

//binning configuration
BinningConfig binConf = new BinningConfig();
binConf.setGridMbr(new double[]{-180, -90, 180, 90});
binConf.setCellHeight(5);
binConf.setCellWidth(5);
binningJob.setBinConf(binConf);

//configure the job
binningJob.configure(job);

//run
boolean success = job.waitForCompletion(true);

public class HBaseRecordInfoProvider implements RecordInfoProvider<ImmutableBytesWritable, Result>, Configurable{
	
	private Result value = null;
	private Configuration conf = null;
	private int srid = 0;

	@Override
	public void setCurrentRecord(ImmutableBytesWritable key, Result value) throws Exception {
		this.value = value;
	}

	@Override
	public String getId() {
		byte[] idb = value.getValue(Bytes.toBytes("other_data"), Bytes.toBytes("user_id"));
		String id = idb != null ? Bytes.toString(idb) : null;
		return id;
	}

	@Override
	public JGeometry getGeometry() {
		byte[] geomb = value.getValue(Bytes.toBytes("location_data"), Bytes.toBytes("geometry"));
		String geomStr = geomb!=null ? Bytes.toString(geomb) : null;
		JGeometry geom = null;
		if(geomStr != null){
			String[] pointsStr = geomStr.split(",");
			geom = JGeometry.createPoint(new double[]{Double.valueOf(pointsStr[0]), Double.valueOf(pointsStr[1])}, 2, srid);
		}
		return geom;
	}

	@Override
	public boolean getExtraFields(Map<String, String> extraFields) {
		byte[] fcb =  value.getValue(Bytes.toBytes("other_data"), Bytes.toBytes("followers_count"));
		if(fcb!=null){
			extraFields.put("followers_count", Bytes.toString(fcb));
		}
		return fcb!=null;
	}

	@Override
	public Configuration getConf() {
		return conf;
	}

	@Override
	public void setConf(Configuration conf) {
		srid = conf.getInt(ConfigParams.SRID, 0);
	}
	
} 

Every time a Vector API job is launched using the command line interface or the web console, a registry file is created for that job. A job registry file contains the following information about the job:

• Job name

• Job ID

• User that executed the job

• Start and finish time

• Parameters used to run the job

• Jobs launched by the first job (called child jobs). Child jobs contain the same fields as the parent job.

A job registry file preserves the parameters used to run the job, which can be used as an aid for running an identical job even when it was not initially run using the command line interface.

By default, job registry files are created under the HDFS path relative to the user folder oracle_spatial/job_registry (for example, /user/hdfs/oracle_spatial/job_registry for the hdfs user).

Job registry files can be removed directly using HDFS commands or using the following utility methods from class oracle.spatial.hadoop.commons.logging.registry.RegistryManager:

• public static int removeJobRegistry(long beforeDate, Configuration conf): Removes all the job registry files that were created before the specified time stamp from the default job registry folder.

• public static int removeJobRegistry(Path jobRegDirPath, long beforeDate, Configuration conf): Removes all the job registry files that were created before the specified time stamp from a specified job registry folder.

The table lists some running times for jobs built using the Vector Analysis API. The jobs were executed using a 4-node cluster. The times may vary depending on the characteristics of the cluster. The test dataset contains over One billion records and the size is above 1 terabyte.

The time taken for the jobs can be decreased by increasing the maximum split size using any of the following configuration parameters.

mapred.max.split.size
mapreduce.input.fileinputformat.split.maxsize

This results in more splits are being processed by each single mapper and improves the execution time. This is done by using the SpatialFilterInputFormat (spatial indexing) or FileSplitInputFormat (spatial hierarchical join, buffer). Also, the same results can be achieved by using the implementation of CombineFileInputFormat as internal InputFormat.

A spatial RDD (Resilient Distributed Dataset) is a Spark RDD that allows you to perform spatial transformations and actions.

The current spatial RDD implementation is the class oracle.spatial.spark.vector.rdd.SpatialJavaRDD for Java and oracle.spatial.spark.vector.scala.rdd.SpatialRDD for Scala. A spatial RDD implementation can be created from an existing instance of RDD or JavaRDD, as shown in the following examples:

Java:

//create a regular RDD
JavaRDD<String> rdd = sc.textFile("someFile.txt");
//create a SparkRecordInfoProvider to extract spatial information from the source RDD’s records
SparkRecordInfoProvider<String> recordInfoProvider = new MySparkRecordInfoProvider();
//create a spatial RDD
SpatialJavaRDD<String> spatialRDD = SpatialJavaRDD.fromJavaRDD(rdd, recordInfoProvider, String.class));

Scala:

//create a regular RDD
val rdd: RDD[String] = sc.textFile("someFile.txt")
//create a SparkRecordInfoProvider to extract spatial information from the source RDD’s records
val recordInfoProvider: SparkRecordInfoProvider[String] = new MySparkRecordInfoProvider()
//create a spatial RDD
val spatialRDD: SpatialRDD[String] = SpatialRDD(rdd, recordInfoProvider)

A spatial RDD takes an implementation of the interface oracle.spatial.spark.vector.SparkRecordInfoProvider, which is used for extracting spatial information from each RDD element.

A regular RDD can be transformed into a spatial RDD of the same generic type, that is, if the source RDD contains records of type String. The spatial RDD will also contain String records.

You can also create a Spatial RDD with records of type oracle.spatial.spark.vector.SparkRecordInfo. A SparkRecordInfo is an abstraction of a record from the source RDD; it holds the source record’s spatial information and may contain a subset of the source record’s data.

The following examples show how to create an RDD of SparkRecordInfo records.

Java:

//create a regular RDD
JavaRDD<String> rdd = sc.textFile("someFile.txt");
//create a SparkRecordInfoProvider to extract spatial information from the source RDD’s records
SparkRecordInfoProvider<String> recordInfoProvider = new MySparkRecordInfoProvider();
//create a spatial RDD
SpatialJavaRDD<SparkRecordInfo> spatialRDD = SpatialJavaRDD.fromJavaRDD(rdd, recordInfoProvider));

Scala:

//create a regular RDD
val rdd: RDD[String] = sc.textFile("someFile.txt")
//create a SparkRecordInfoProvider to extract spatial information from the source RDD’s records
val recordInfoProvider: SparkRecordInfoProvider[String] = new MySparkRecordInfoProvider()
//create a spatial RDD
val spatialRDD: SpatialRDD[SparkRecordInfo] = SpatialRDD.fromRDD(rdd, recordInfoProvider))

A spatial RDD of SparkRecordInfo records has the advantage that spatial information does not need to be extracted from each record every time it is needed for a spatial operation.

You can accelerate spatial searches by spatially indexing a spatial RDD. Spatial indexing is described in section Spatially Indexing a Spatial RDD.

The spatial RDD provides the following spatial transformations and actions, which are described in the sections Spatial Transformations and Spatial Actions (MBR and NearestNeighbors).

Spatial transformations:

• filter

• flatMap

• join (available when creating a spatial index)

Spatial Actions:

• MBR

• nearestNeighbors

public class CSVRecordInfoProvider implements SparkRecordInfoProvider<String>{
    private int srid = 8307;

    //receives an RDD record and fills the given recordInfo
    public boolean getRecordInfo(String record, SparkRecordInfo recordInfo) {
        try {
            String[] tokens = record.split(",");
            //expected records have the format: id,name,last_name,x,y where x and y are optional 
            //output recordInfo will contain the fields id, last name and geometry
            recordInfo.addField("id", tokens[0]);
            recordInfo.addField("last_name", tokens[2]);
            if (tokens.length == 5) {
                recordInfo.setGeometry(JGeometry.createPoint(tokens[3], tokens[4], 2, srid));
            }
        } catch (Exception ex) {
            //return false when there is an error extracting data from the input value
            return false;
        }
        return true;
    }

    public void setSrid(int srid) {this.srid = srid;}	
    public int getSrid() {return srid;}
}

The transformations described in the following subtopics are available for spatial RDD, spatial pair RDD, and the distributed spatial index unless stated otherwise (for example, a join transformation is only available for a distributed spatial index).

• Filter Transformation
  
• FlatMap Transformation
  
• Join Transformation
  
• Controlling Spatial Evaluation
  
• Spatially Enabled Transformations
  

Spatial RDDs,spatial pair RDDs, and the distributed spatial index provide the following spatial actions.

• MBR: Calculates the RDD’s minimum bounding rectangle (MBR). The MBR is only calculated once and cached so the second time it is called, it will not be recalculated. The following examples show how to get the MBR from a spatial RDD. (This transformation is not available for DistributedSpatialIndex.)

  Java:

  doubl[] mbr = spatialRDD.getMBR();

  Scala:

  val mbr: Array[Double] = spatialRDD.getMBR()

• NearestNeighbors: Returns a list containing the K nearest elements from an RDD or distributed spatial index to a given geometry. Additionally, a user-defined filter lambda function can be passed, so that only the records that pass the filter will be candidates to be part of the K nearest neighbors list. The following examples show how to get the 5 records closest to the given point.

  Java:

  JGeometry qryWindow = JGeometry.createPoint(new double[] { 2.0, 1.0 }, 2, srid));
  SpatialOperationConfig soc = new SpatialOperationConfig(SpatialOperation.None, qryWindow, 0.05);
  List<SparkRecordInfo> nearestNeighbors = spatialRDD.nearestNeighbors(
  (record)->{
  	return ((Integer)record.getField(“followers_count”))>1000;
  }, 5, soc);
  

  Scala:

  val qryWindow: JGeometry = JGeometry.createPoint(Array(2.0, 1.0 ), 2, srid))
  val soc: SpatialOperationConfig = new SpatialOperationConfig(SpatialOperation.None, qryWindow, 0.05)
  val nearestNeighbors: Seq[SparkRecordInfo] = spatialRDD.nearestNeighbors(
  record=>{ record.getField(“followers_count”).asInstanceOf[Int]>1000 }, 5, soc);
  

A spatial RDD can be spatially indexed to speed up spatial searches when performing spatial transformations.

A spatial index repartitions the spatial RDD so that each partition only contains records on some specific area. This allows partitions that do not contain results in a spatial search to be quickly discarded, making the search faster.

A spatial index is created through the Java abstract class oracle.spatial.spark.vector.index.DistributedSpatialIndex or its Scala equivalent oracle.spatial.spark.vector.scala.index.DistributedSpatialIndex, both of which use a specific implementation to create the actual spatial index. The following examples show how to create a spatial index using a QuadTree-based spatial index implementation.

Java:

DistributedSpatialIndex<String> index = DistributedSpatialIndex.createIndex(sparkContext, spatialRDD1, new QuadTreeConfiguration());

Scala:

val index: DistributedSpatialIndex[String] = DistributedSpatialIndex.createIndex(spatialRDD1, new QuadTreeConfiguration())(sparkContext)

The type of spatial index implementation is determined by the last parameter, which is a subtype of oracle.spatial.spark.vector.index.SpatialPartitioningConfiguration. Depending on the index implementation, the configuration parameter may accept different settings for performing partitioning and indexing. Currently, the only implementation of a spatial index is the class oracle.spatial.spark.vector.index.quadtree.QuadTreeDistIndex, and it receives a configuration of type oracle.spatial.spark.vector.index.quadtree.QuadTreeConfiguration.

The DistributedSpatialIndex class currently supports the filter, flatMap, join, and nearestNeighbors transformations, which are described in Spatial Transformations.

A spatial index can be persisted using the method DistributedSpatialIndex.save(), which takes an existing SparkContext and a path where the index will be stored. The path may be in a local or a distributed (HDFS) file system. Similarly, a persisted spatial index can be loaded by calling the method DistributedSpatialIndex.load(), which also takes an existing SparkContext and the path where the index is stored.

• Spatial Partitioning of a Spatial RDD
  
• Local Spatial Indexing of a Spatial RDD
  

A Spatial DStream is a Spark DStream that allows spatial transformations to be performed.

The current Spatial DStream implementations are the class oracle.spatial.spark.vector.streaming.dstream.SpatialJavaDStream and oracle.spatial.spark.vector.streaming.dstream.SpatialJavaPairDStream for Java, and oracle.spatial.spark.vector.scala.streaming.dstream.SpatialDStream for Scala. A spatial DStream can be created from an existing instance of DStream or JavaDStream, as shown in the following examples.

Java:

//create a regular DStram
JavaDStream<String> stream = ssc.socketTextStream(host, port);
//create a SparkRecordInfoProvider to extract spatial information from the stream
SparkRecordInfoProvider<String> recordInfoProvider = new TextRecordInfoProvider();
//create a Spatial DStream
SpatialJavaDStream<SparkRecordInfo> spatialStream = SpatialJavaDStream.fromJavaDStream(stream, recordInfoProvider);

Scala:

//create a regular RDD
val stream: DStream[String] = ssc.socketTextStream(host, port)
//create a SparkRecordInfoProvider to extract spatial information from the stream
val recordInfoProvider: SparkRecordInfoProvider[String] = new TextRecordInfoProvider()
//create a Spatial DStream
val spatialStream: SpatialDStream[SparkRecordInfo] = SpatialDStream.fromDStream(stream, recordInfoProvider)

A Spatial DStream takes an implementation of the interface oracle.spatial.spark.vector.SparkRecordInfoProvider, which is used for extracting spatial information from each element contained in the stream.

A regular DStream can be transformed into a Spatial DStream of the same generic type; that is, if the source DStream contains records of type String, the Spatial DStream will also contain String records. You can also create a Spatial DStream with records of type oracle.spatial.spark.vector.SparkRecordInfo. A SparkRecordInfo is an abstraction of a record from the source DStream; it holds the source record’s spatial information and may contain a subset of the source record’s data. The following examples show how to create a Spatial DStream of SparkRecordInfo records.

Java:

//create a regular DStram
JavaDStream<String> stream = ssc.socketTextStream(host, port);
//create a SparkRecordInfoProvider to extract spatial information from the stream
SparkRecordInfoProvider<String> recordInfoProvider = new TextRecordInfoProvider();
//create a Spatial DStream
SpatialJavaDStream<SparkRecordInfo> spatialStream = SpatialJavaDStream.fromJavaDStream(stream, recordInfoProvider);

Scala:

//create a regular RDD
val stream: DStream[String] = ssc.socketTextStream(host, port)
//create a SparkRecordInfoProvider to extract spatial information from the stream
val recordInfoProvider: SparkRecordInfoProvider[String] = new TextRecordInfoProvider()
//create a Spatial DStream
val spatialStream: SpatialDStream[SparkRecordInfo] = SpatialDStream.fromDStream(stream, recordInfoProvider)

A Spatial DStream of SparkRecordInfo records has the advantage that spatial information does not need to be extracted from each record every time it is needed for a spatial operation.

The Spatial DStream provides the following spatial transformations, which are available for both the Java and Scala Spatial DStream implementations.

• Filter Transformation (Spatial DStream)
  
• FlatMap Transformation (Spatial DStream)
  
• NearestNeighbors Transformation (Spatial DStream)
  
• Enrich Transformation (Spatial DStream)
  

The Spark Vector API provides utilities to easily read data from common spatial formats such as GeoJSON and ESRI ShapeFile.

The Java class oracle.spatial.spark.vector.io.SpatialSources and the Scala class oracle.spatial.spark.vector.scala.io.SpatialSources contain static methods to read data from GeoJSON and ShapeFile formats by specifying the data path, the data Spatial Reference System ID (SRID), and the list of non-spatial fields to be loaded.

The following examples show how to load data from a GeoJSON file. The records are automatically transformed to instances of SparkRecordInfo, which contain the spatial information plus the _id and followers_count fields. If all the fields need to be retrieved, null can be passed instead of the whole list of fields. Both GeoJSON and Shapefile read methods contain an overload that returns the original records as String and MapWritable representations, respectively.

Java:

//list of GeoJSON field names to be loaded for each feature
List<String> fieldNames = new ArrayList<String>();
fieldNames.add("_id");
fieldNames.add("followers_count");

//create a spatial RDD from a GeoJSON file
SpatialJavaRDD<SparkRecordInfo> spatialRDD = SpatialSources.readGeoJSONRecordInfo(geoJSONInputPath, 8307, fieldNames, sparkContext);

Scala:

//create a spatial RDD from a GeoJSON file
val spatialRDD = SpatialSources.readGeoJSONRecordInfo(geoJSONInputPath, 8307, Seq("_id","followers_count"))(sparkContext)

Or, using implicit classes:

//create a spatial RDD from a GeoJSON file
import oracle.spatial.spark.vector.scala.io.SpatialSources.ImplicitSpatialSources
val spatialRDD = sparkContext.readGeoJSONRecordInfo(geoJSONInputPath, 8307, Seq("_id","followers_count"))

The Spatial Spark SQL API supports Spark SQL DataFrame objects containing spatial information in any format.

Oracle Big Data Spatial Vector Hive Analysis can be used with Spark SQL.

//create HiveContext
HiveContext sqlContext = new HiveContext(sparkContext.sc());
//get the spatial DataFrame from the SpatialRDD
//the geometries are in GeoJSON format
DataFrame spatialDataFrame = spatialRDD.createSpatialDataFrame(sqlContext, properties);
// Register the DataFrame as a table.
spatialDataFrame.registerTempTable("tweets");
//register UDFs
sqlContext.sql("create temporary function ST_Polygon as 'oracle.spatial.hadoop.vector.hive.ST_Polygon'");
sqlContext.sql("create temporary function ST_Point as 'oracle.spatial.hadoop.vector.hive.ST_Point'");
sqlContext.sql("create temporary function ST_Contains as 'oracle.spatial.hadoop.vector.hive.function.ST_Contains'");
// SQL can be run over RDDs that have been registered as tables.
StringBuffer query = new StringBuffer();
query.append("SELECT geometry, friends_count, location, followers_count FROM tweets ");
query.append("WHERE ST_Contains( ");
query.append("	ST_Polygon('{\"type\": \"Polygon\",\"coordinates\": [[[-106, 25], [-106, 30], [-104, 30], [-104, 25], [-106, 25]]]}', 8307) ");
query.append("	, ST_Point(geometry, 8307) ");
query.append("	, 0.05)");
query.append("	and followers_count > 50");
DataFrame results = sqlContext.sql(query.toString());		
//Filter the tweets in a query window (somewhere in the north of Mexico) 
//and with more than 50 followers.
//Note that since the geometries are in GeoJSON format it is possible to create the ST_Point like
//ST_Point(geometry, 8307)
//instead of
//ST_Point(geometry, 'oracle.spatial.hadoop.vector.hive.json.GeoJsonHiveRecordInfoProvider')
List<String> filteredTweets = results.javaRDD().map(new Function<Row, String>() {
  public String call(Row row) {
	  StringBuffer sb = new StringBuffer();
	  sb.append("Geometry: ");
	  sb.append(row.getString(0));
	  
	  sb.append("\nFriends count: ");
	  sb.append(row.getString(1));
	  sb.append("\nLocation: ");
	  sb.append(row.getString(2));
	  sb.append("\nFollowers count: ");
	  sb.append(row.getString(3));
	  return sb.toString();
  }
}).collect();
//print the filtered tweets
filteredTweets.forEach(tweet -> System.out.println("Tweet: "+tweet)); 

• Spark 2 API Enhancements
  
• Spatial Analysis Spark SQL UDFs
  

The Java API provides two ways to generate results based on a spatial index that can be rendered on maps using the Oracle Map API.

• Render all the records or filtered records as an image. Some examples of the possibilities are:
  • Render an image using a numerical attribute and setting the minimum and maximum values.
  • Render an image using a numerical attribute and setting limits with defined colors.
  • Render an image using a textual attribute and setting possible values with defined colors.
• Render the records as a categorization result (such as by countries or regions). Some examples of the possibilities are:
  • Categorize by country using a numerical attribute and setting minimum and maximum values, using default colors and the average or sum as categorization operation.
  • Categorize using the count operation and setting limits and colors.
  • Categorize using the count operation, the default colors, and the automatic limits detection.
  • Categorize using textual attribute and setting possible values and colors.

Detailed code examples can be found under the folder /opt/oracle/oracle-spatial-graph/spatial/vector/examples/.

Oracle Database data can be used as the data source of a Spatial RDD by using the Spark Vector Analysis API.

The class oracle.spatial.spark.vector.util.JDBCUtils (or oracle.spatial.spark.vector.scala.util.JDBCUtils for Scala) provides convenience methods for creating a Spatial RDD from an Oracle database table or from a SQL query to an Oracle database. The table or SQL query should contain one column of type SDO_GEOMETRY in order to create a Spatial RDD.

Both the from-table and from-query method versions require a connection to the Oracle database, which is supplied by a lambda function defined by the template oracle.spatial.spark.vector.util.ConnectionSupplier (or oracle.spatial.spark.vector.scala.util.ConnectionSupler for Scala).

The resulting Spatial RDD type parameter will always be SparkRecordInfo, that is, the resulting RDD will contain records of the type SparkRecordInfo, which will contain the fields specified when querying the table or the columns in the SELECT section of the SQL query. By default, the name and type of the columns retrieved are inferred using the ResultSet metadata; however, you can control the naming and type of the retrieved fields by supplying an implementation of SparkRecordInfoProvider

The following examples show how to create a Spatial RDD from a table and from a SQL query respectively.

SpatialJavaRDD<SparkRecordInfoProvider> jdbcSpatialRDD = JDBCUtils.createSpatialRDDFromTable(
	sparkContext, //spark context
	()->{
		Class.forName("oracle.jdbc.driver.OracleDriver");  
		return new DriverManager.getConnection(connURL, usr, pwd);
	}, //DB connection supplier lambda
	“VEHICLES”, //DB table
	Arrays.asList(new String[]{"ID","DESC","LOCATION"}), //list of fields to retrieve
	null //SparkRecordInfoProvider<ResultSet, SparkRecordIngo> (optional)
);

SpatialJavaRDD<SparkRecordInfoProvider> jdbcSpatialRDD = JDBCUtils.createSpatialRDDFromQuery(
	sparkContext, //spark context
	()->{
		Class.forName("oracle.jdbc.driver.OracleDriver");  
		return new DriverManager.getConnection(connURL, usr, pwd);
	}, //DB connection supplier lambda
	“SELECT * FROM VEHICLES WHERE category > 5”, //SQL query
	null //SparkRecordInfoProvider<ResultSet, SparkRecordIngo> (optional)
);

A record in a Hive table may contain a geometry field in any format like JSON, WKT, or a user-specifiedformat. Geometry constructors like ST_Geometry can create a geometry receiving the GeoJSON, WKT, or WKB representation of the geometry. If the geometry is stored in another format, a HiveRecordInfoProvider can be used.

HiveRecordInfoProvider is a component that interprets the geometry field representation and returns the geometry in a GeoJSON format.

The returned geometry must contain the geometry SRID, as in the following example format:

{"type":<geometry-type", "crs": {"type": "name", "properties": {"name": "EPSG:4326"}}"coordinates":[c1,c2,....cn]} 

The HiveRecordInfoProvider interface has the following methods:

• void setCurrentRecord(Object record)

• String getGeometry()

The method setCurrentRecord() is called by passing the current geometry field provided when creating a geometry in Hive. The HiveRecordInfoProvider is used then to get the geometry or to return null if the record has no spatial information.

The information returned by the HiveRecordInfoProvider is used by the Hive Spatial functions to create geometries (see Hive and Spark Spatial SQL Functions).

{"longitude":-71.46, "latitude":42.35}

public class SimpleHiveRecordInfoProvider implements HiveRecordInfoProvider{
  private static final Log LOG =
    LogFactory.getLog(SimpleHiveRecordInfoProvider.class.getName());

  private JsonNode recordNode = null;
  private ObjectMapper jsonMapper = null;

  public SimpleHiveRecordInfoProvider(){
    jsonMapper = new ObjectMapper();
  }
  
  @Override
  public void setCurrentRecord(Object record) throws Exception {
    try{
      if(record != null){
        //parse the current value
        recordNode = jsonMapper.readTree(record.toString());
      }
    }catch(Exception ex){
      recordNode = null;
      LOG.warn("Problem reading JSON record
        value:"+record.toString(), ex);
    }    
  }

  @Override
  public String getGeometry() {
    if(recordNode == null){
      return null;
    }
    
    JGeometry geom = null;

    try{
      geom = JsonUtils.readGeometry(recordNode, 
          2, //dimensions
          8307 //SRID
          );
    }catch(Exception ex){
      recordNode = null;
      LOG.warn("Problem reading JSON record
        geometry:"+recordNode.toString(), ex);
    }
    
    if(geom != null){
      StringBuilder res = new StringBuilder();
      //Get a GeoJSON representation of the JGeometry
      GeoJsonGen.asGeometry(geom, res);
      String result = res.toString();
      //add SRID to GeoJSON and return the result
      return JsonUtils.addSRIDToGeoJSON(result, 8307);
    }
    
     return null;
  }
}

The Hive Spatial API consists of Oracle-supplied Hive User Defined Functions that can be used to create geometries and perform operations using one or two geometries.

The functions can be grouped into logical categories: types, single-geometry, and two-geometries. (Hive and Spark Spatial SQL Functions lists the functions in each category and provides reference information about each function.)

add jar
  /opt/oracle/oracle-spatial-graph/spatial/vector/jlib/ojdbc8.jar
  /opt/oracle/oracle-spatial-graph/spatial/vector/jlib/sdoutl.jar
  /opt/oracle/oracle-spatial-graph/spatial/vector/jlib/sdoapi.jar
  /opt/oracle/oracle-spatial-graph/spatial/vector/jlib/sdohadoop-vector.jar
  /opt/oracle/oracle-spatial-graph/spatial/vector/jlib/sdohadoop-vector-hive.jar;

create temporary function ST_Point as 'oracle.spatial.hadoop.vector.hive.ST_Point';
create temporary function ST_Polygon as 'oracle.spatial.hadoop.vector.hive.ST_Polygon';
create temporary function ST_Contains as 'oracle.spatial.hadoop.vector.hive.function.ST_Contains';
 
CREATE EXTERNAL TABLE IF NOT EXISTS sample_tweets (id STRING, geometry STRING, followers_count STRING, friends_count STRING, location STRING)                                         
ROW FORMAT SERDE 'oracle.spatial.hadoop.vector.hive.json.GeoJsonSerDe'              
STORED AS INPUTFORMAT 'oracle.spatial.hadoop.vector.geojson.mapred.GeoJsonInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION '/user/oracle/twitter';


SELECT id, followers_count, friends_count, location FROM sample_tweets
WHERE 
ST_Contains(
  ST_Polygon(
    '{"type": "Polygon",
    "coordinates": 
      [[[-106, 25],[-106, 30], [-104, 30], [-104, 25], [-106, 25]]]}', 
    8307
  ), 
  ST_Point(geometry, 8307), 
  0.5
)
and followers_count > 50;

Hive spatial queries can use a previously created spatial index, which you can create using the Java API (see Spatial Indexing).

If you do not need to use the index in API functions that will access the original data, you can specify isMapFileIndex=false when you call oracle.spatial.hadoop.vector.mapred.job.SpatialIndexing, or you can use the function setMapFileIndex(false). In these cases, the index will have the following structure:

HDFSIndexDirectory/part-xxxxx

And in these cases, when creating a Hive table, just provide the folder where you created the index.

If you need to access the original data and you do not set the parameter isMapFileIndex=false, the index structure is as follows:

part-xxxxx
  data
  index

In such cases, to create a Hive table, the data files of the index are needed. Copy the data files into a new HDFS folder, with each data file having a different name, like data1, data2,, and so on. The new folder will be used to create the Hive table.

The index contains the geometry records and extra fields. That data can be used when creating the Hive table.

(Note that Spatial Indexing Class Structure describes the index structure, and RecordInfoProvider provides an example of a RecordInfoProvider adding extra fields.)

InputFormat oracle.spatial.hadoop.vector.mapred.input.SpatialIndexTextInputFormat will be used to read the index. The output of this InputFormat is GeoJSON.

Before running any query, you can specify a minimum bounding rectangle (MBR) that will perform a first data filtering using SpatialIndexTextInputFormat..

add jar
  /opt/oracle/oracle-spatial-graph/spatial/vector/jlib/ojdbc8.jar
  /opt/oracle/oracle-spatial-graph/spatial/vector/jlib/sdoutl.jar
  /opt/oracle/oracle-spatial-graph/spatial/vector/jlib/sdoapi.jar
  /opt/oracle/oracle-spatial-graph/spatial/vector/jlib/sdohadoop-vector.jar
  /opt/oracle/oracle-spatial-graph/spatial/vector/jlib/sdohadoop-vector-hive.jar;

create temporary function ST_Polygon as 'oracle.spatial.hadoop.vector.hive.ST_Polygon';
create temporary function ST_Point as 'oracle.spatial.hadoop.vector.hive.ST_Point';
create temporary function ST_Contains as 'oracle.spatial.hadoop.vector.hive.function.ST_Contains';

set oracle.spatial.boundaries=-180,-90,180,90;
set oracle.spatial.index.includedExtraFields=followers_count,friends_count,location;

CREATE EXTERNAL TABLE IF NOT EXISTS sample_tweets_index (id STRING, geometry STRING, followers_count STRING, friends_count STRING, location STRING)                                         
ROW FORMAT SERDE 'oracle.spatial.hadoop.vector.hive.json.GeoJsonSerDe'              
STORED AS INPUTFORMAT 'oracle.spatial.hadoop.vector.mapred.input.SpatialIndexTextInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION '/user/oracle/twitter/index';

set oracle.spatial.spatialQueryWindow={"type": "Polygon","coordinates": [[[-107, 24], [-107, 31], [-103, 31], [-103, 24], [-107, 24]]]};

SELECT id, followers_count, friends_count, location FROM sample_tweets
WHERE ST_Contains(
  ST_Polygon('{"type": "Polygon","coordinates": [[[-106, 25], [-106, 30], [-104, 30], [-104, 25], [-106, 25]]]}', 8307)
  , ST_Point(geometry, 8307)
  , 0.5)
  and followers_count > 50;

1. Open http://<oracle_big_data_spatial_vector_console>:8045/spatialviewer/?root=vector.
2. Click Categorization, then Results.
3. Click any one of the Templates. For example, World Continents.

   The World Continents template is displayed.

4. Click any one of the Results displayed.

   Different continents appear with different patches of colors.

5. Click any continent from the map. For example, North America.

   The template changes to World Countries and the focus changes to North America with the results by country.

1. Click Categorization, then Results.
2. Select a Template .
3. Click the icon for saving the results.
4. Specify a Name.
5. Click Choose File to select the File location.
6. Click Save.

   The results can be located in the folder clustering_results contained in the SpatialViewer local working directory (see Configuring SpatialViewer on Oracle Big Data Appliance).

1. Add the template JSON file in the folder /opt/oracle/oracle-spatial-graph/spatial/web-server/spatialviewer/templates/.
2. Add the template configuration file in the folder /opt/oracle/oracle-spatial-graph/spatial/web-server/spatialviewer/templates/_config_.

Each template has a configuration file. The template configuration files are located in the folder /opt/oracle/oracle-spatial-graph/spatial/web-server/spatialviewer/templates/_config_. The name of the configuration file is the same as the template files suffixed with config.json instead of .json.For example, the configuration file name of the template file usa_states.json is usa_states.config.json. The configuration parameters are:

• name: Name of the template to be shown on the console. For example, name: USA States.

• display_attribute: When displaying a categorization result, a cursor move on the top of a feature displays this property and result of the feature. For example, display_attribute: STATE NAME.

• point_geometry: True, if the template contains point geometries and false, in case of polygons. For example, point_geometry: false.

• child_templates (optional): The templates that can have several possible child templates separated by a coma. For example, child_templates: ["world_states_provinces, usa_states(properties.COUNTRY CODE:properties.PARENT_REGION)"].

  If the child templates do not specify a linked field, it means that all the features inside the parent features are considered as child features. In this case, the world_states_provinces doesn't specify any fields. If the link between parent and child is specified, then the spatial relationship doesn't apply and the feature properties link are checked. In the above example, the relationship with the usa_states is found with the property COUNTRY CODE in the current template, and the property PARENT_REGION in the template file usa_states.json.

• srid: The SRID of the template's geometries. For example, srid: 8307.

• back_polygon_template_file_name (optional): A template with polygon geometries to set as background when showing the defined template. For example, back_polygon_template_file_name: usa_states.

• vectorLayers: Configuration specific to the MVSuggest service. For example:

  {
  "vectorLayers": [
  {
  		"gnidColumns":["_GNID"],
  		"boostValues":[2.0,1.0,1.0,2.0]
   	}
   	]
   }
  

  Where:

  • gnidColumns is the name of the column(s) within the Json file that represents the Geoname ID. This value is used to support multiple languages with MVSuggest. (See references of that value in the file templates/_geonames_/alternateNames.json.) There is nodefault value for this property.

  • boostValues is an array of float numbers that represent how important a column is within the "properties" values for a given row. The higher the number, the more important that field is. A value of zero means the field will be ignored. When boostValues is not present, all fields receive a default value of 1.0, meaning they all are equally important properties. The MVSuggest service may return different results depending on those values. For a Json file with the following properties, the boost values might be as follows:

    "properties":{"Name":"New York City","State":"NY","Country":"United States","Country Code":"US","Population":8491079,"Time Zone":"UTC-5"}
    "boostValues":[3.0,2.0,1.0,1.0,0.0,0.0]
    

1. Open http://<oracle_big_data_spatial_vector_console>:8045/spatialviewer/?root=vector.
2. Click Clustering, then Results.
3. Click any one of the Results displayed.

1. Open http://<oracle_big_data_spatial_vector_console>:8045/spatialviewer/?root=vector.
2. Click Clustering, then Results.
3. Click the icon for saving the results.
4. Specify a name.
5. Specify the SRID of the geometries. For example: 8307
6. Click Choose File and select the file location.
7. Click Save.

1. Open http://<oracle_big_data_spatial_vector_console>:8045/spatialviewer/?root=vector.
2. Click Binning, then Results.
3. Click any of the Results displayed.

1. Open http://<oracle_big_data_spatial_vector_console>:8045/spatialviewer/?root=vector.
2. Click Binning, then View Results.
3. Click the icon for saving the results.
4. Specify the SRID of the geometries. For example: 8307
5. Specify the thematic attribute, which must be a property of the features in the result. For example, the count attribute can be used to create results depending on the number of results per bin.
6. Click Choose File and select the file location.
7. Click Save.

To create a spatial index, use a command in the following format:

hadoop jar <HADOOP_LIB_PATH>/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.SpatialIndexing [generic options] input=<path|comma_separated_paths|path_pattern> output=<path> inputFormat=<InputFormat_subclass> recordInfoProvider=<RecordInfoProvider_subclass> [srid=<integer_value>] [geodetic=<true|false>] [tolerance=<double_value>] [boundaries=<minX,minY,maxX,maxY>] [indexName=<index_name>] [indexMetadataDir=<path>] [overwriteIndexMetadata=<true|false>] [  mvsLocation=<path|URL> [mvsMatchLayers=<comma_separated_layers>][mvsMatchCountry=<country_name>][mvsSpatialResponse=<[NONE, FEATURE_GEOMETRY, FEATURE_CENTROID]>][mvsInterfaceType=<LOCAL, WEB>][mvsIsRepository=<true|false>][rebuildMVSIndex=<true|false>][mvsPersistentLocation=<hdfs_path>][mvsOverwritePersistentLocation=<true|false>] ]

To use the new Hadoop API format, replace oracle.spatial.hadoop.vector.mapred.job.SpatialIndexing with oracle.spatial.hadoop.vector.mapreduce.job.SpatialIndexing.

Input/output arguments:

• input : the location of the input data. It can be expressed as a path, a comma separated list of paths, or a regular expression.

• inputFormat: the inputFormat class implementation used to read the input data.

• recordInfoProvider: the recordInfoProvider implementation used to extract information from the records read by the InputFormat class.

• output: the path where the spatial index will be stored

Spatial arguments:

• srid (optional, default=0): the spatial reference system (coordinate system) ID of the spatial data.

• geodetic (optional, default depends on the srid): boolean value that indicates whether the geometries are geodetic or not.

• tolerance (optional, default=0.0): double value that represents the tolerance used when performing spatial operations.

• boundaries (optional, default=unbounded): the minimum and maximum values for each dimension, expressed as comma separated values in the form: minX,minY,maxX,maxY

Spatial index metadata arguments:

• indexName (optional, default=output folder name):The name of the index to be generated.

• indexMetadataDir (optional, default=hdfs://server:port/user/<current_user>/oracle_spatial/index_metadata/): the directory where the spatial index metadata will be stored.

• overwriteIndexMetadata (optional, default=false) boolean argument that indicates whether the index metadata can be overwritten if an index with the same name already exists.

MVSuggest arguments:

• mvsLocation: The path to the MVSuggest directory or repository for local standalone instances of MVSuggest or the service URL when working with a remote instance. This argument is required when working with MVSuggest.

• mvsMatchLayers (optional, default=all): comma separated list of layers. When provided, MVSuggest will only use these layers to perform the search.

• mvsMatchCountry (optional, default=none): a country name which MVSuggest will give higher priority when performing matches.

• mvsSpatialResponse (optional, default=CENTROID): the type of the spatial results contained in each returned match. It can be one of the following values: NONE, FEATURE_GEOMETRY, FEATURE_CENTROID.

• mvsInterfaceType (optional: default=LOCAL): the type of MVSuggest service used, it can be LOCAL or WEB.

• mvsIsRepository (optional: default=false) (LOCAL only): boolean value which specifies whether mvsLocation points to a whole MVS directory(false) or only to a repository(true). An MVS repository contains only JSON templates; it may or not contain a _config_ and _geonames_ folder.

• mvsRebuildIndex (optional, default=false)(LOCAL only):boolean value specifying whether the repository index should be rebuilt or not.

• mvsPersistentLocation (optional, default=none)(LOCAL only): an HDFS path where the MVSuggest directory will be saved.

• mvsIsOverwritePersistentLocation (optional, default=false): boolean argument that indicates whether an existing mvsPersistentLocation must be overwritten in case it already exists.

Example: Create a spatial index called indexExample. The index metadata will be stored in the HDFS directory spatialMetadata.

hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop/lib/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.SpatialIndexing input="/user/hdfs/demo_vector/tweets/part*" output=/user/hdfs/demo_vector/tweets/spatial_index inputFormat=oracle.spatial.hadoop.vector.geojson.mapred.GeoJsonInputFormat recordInfoProvider=oracle.spatial.hadoop.vector.geojson.GeoJsonRecordInfoProvider srid=8307 geodetic=true tolerance=0.5 indexName=indexExample indexMetadataDir=indexMetadataDir overwriteIndexMetadata=true

Example: Create a spatial index using MVSuggest to assign a spatial location to records that do not contain geometries.

hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop/lib/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.SpatialIndexing input="/user/hdfs/demo_vector/tweets/part*" output=/user/hdfs/demo_vector/tweets/spatial_index inputFormat=oracle.spatial.hadoop.vector.geojson.mapred.GeoJsonInputFormat recordInfoProvider=mypackage.Simple LocationRecordInfoProvider srid=8307 geodetic=true tolerance=0.5 indexName=indexExample indexMetadataDir=indexMetadataDir overwriteIndexMetadata=true mvsLocation=file:///local_folder/mvs_dir/oraclemaps_pub/ mvsRepository=true

To create a categorization result, use a command in one of the following formats.

With a Spatial Index

hadoop jar <HADOOP_LIB_PATH >/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.Categorization [generic options] ( indexName=<indexName> [indexMetadataDir=<path>] )  | ( input=<path|comma_separated_paths|path_pattern> isInputIndex=true [srid=<integer_value>] [geodetic=<true|false>] [tolerance=<double_value>] [boundaries=<min_x,min_y,max_x,max_y>]  ) output=<path> hierarchyIndex=<hdfs_hierarchy_index_path> hierarchyInfo=<HierarchyInfo_subclass> [hierarchyDataPaths=<level1_path,level2_path,,levelN_path>] spatialOperation=<[None, IsInside, AnyInteract]>

Without a Spatial Index

hadoop jar <HADOOP_LIB_PATH >/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.Categorization [generic options] input=<path|comma_separated_paths|path_pattern> inputFormat=<InputFormat_subclass> recordInfoProvider=<RecordInfoProvider_subclass> [srid=<integer_value>] [geodetic=<true|false>] [tolerance=<double_value>] [boundaries=<min_x,min_y,max_x,max_y>]  output=<path> hierarchyIndex=<hdfs_hierarchy_index_path> hierarchyInfo=<HierarchyInfo_subclass>  hierarchyDataPaths=<level1_path,level2_path,,levelN_path>] spatialOperation=<[None, IsInside, AnyInteract]>

Using MVSuggest

hadoop jar <HADOOP_LIB_PATH >/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.Categorization [generic options] (indexName=<indexName> [indexMetadataDir=<path>])  | 
( 
(input=<path|comma_separated_paths|path_pattern> isInputIndex=true)  | (input=<path|comma_separated_paths|path_pattern> inputFormat=<InputFormat_subclass> recordInfoProvider=<LocalizableRecordInfoProvider_subclass>)
[srid=<integer_value>] [geodetic=<true|false>] [tolerance=<double_value>] [boundaries=<min_x,min_y,max_x,max_y>]  
) output=<path> 
mvsLocation=<path|URL> [mvsMatchLayers=<comma_separated_layers>] [mvsMatchCountry=<country_name>] [mvsSpatialResponse=<[NONE, FEATURE_GEOMETRY, FEATURE_CENTROID]>] [mvsInterfaceType=<[UNDEFINED, LOCAL, WEB]>] [mvsIsRepository=<true|false>] [mvsRebuildIndex=<true|false>] [mvsPersistentLocation=<hdfs_path>] [mvsOverwritePersistentLocation=<true|false>] [mvsMaxRequestRecords=<integer_number>]  hierarchyIndex=<hdfs_hierarchy_index_path> hierarchyInfo=<HierarchyInfo_subclass>

To use the new Hadoop API format, replace oracle.spatial.hadoop.vector.mapred.job.Categorization with oracle.spatial.hadoop.vector.mapreduce.job.Categorization.

Input/output arguments:

• indexName: the name of an existing spatial index. The index information will be looked at the path given by indexMetadataDir. When used, the argument input is ignored.

• indexMetadataDir (optional, default=hdfs://server:port/user/<current_user>/oracle_spatial/index_metadata/): the directory where the spatial index metadata is located

• input : the location of the input data. It can be expressed as a path, a comma separated list of paths, or a regular expression. (Ignored if indexName is specified.)

• inputFormat: the inputFormat class implementation used to read the input data. (Ignored if indexName is specified.)

• recordInfoProvider: the recordInfoProvider implementation used to extract information from the records read by the InputFormat class. (Ignored if indexName is specified.)

• output: the path where the spatial index will be stored

Spatial arguments:

• srid (optional, default=0): the spatial reference system (coordinate system) ID of the spatial data.

• geodetic (optional, default depends on the srid): boolean value that indicates whether the geometries are geodetic or not.

• tolerance (optional, default=0.0): double value that represents the tolerance used when performing spatial operations.

• boundaries (optional, default=unbounded): the minimum and maximum values for each dimension, expressed as comma separated values in the form: minX,minY,maxX,maxY

• spatialOperation: the spatial operation to perform between the input data set and the hierarchical data set. Allowed values are IsInside and AnyInteract.

Hierarchical data set arguments:

• hierarchyIndex: the HDFS path of an existing hierarchical index or where it can be stored if it needs to be generated.

• hierarchyInfo: the fully qualified name of a HierarchyInfo subclass which is used to describe the hierarchical data.

• hierarchyDataPaths (optional, default=none): a comma separated list of paths of the hierarchy data. The paths should be sorted in ascending way by hierarchy level. If a hierarchy index path does not exist for the given hierarchy data, this argument is required.

MVSuggest arguments:

• mvsLocation: The path to the MVSuggest directory or repository for local standalone instances of MVSuggest or the service URL when working with a remote instance. This argument is required when working with MVSuggest.

• mvsMatchLayers (optional, default=all): comma separated list of layers. When provided, MVSuggest will only use these layers to perform the search.

• mvsMatchCountry (optional, default=none): a country name which MVSuggest will give higher priority when performing matches.

• mvsSpatialResponse (optional, default=CENTROID): the type of the spatial results contained in each returned match. It can be one of the following values: NONE, FEATURE_GEOMETRY, FEATURE_CENTROID.

• mvsInterfaceType (optional: default=LOCAL): the type of MVSuggest service used, it can be LOCAL or WEB.

• mvsIsRepository (optional: default=false) (LOCAL only): Boolean value that specifies whether mvsLocation points to a whole MVS directory(false) or only to a repository(true). An MVS repository contains only JSON templates; it may or not contain a _config_ and _geonames_ folder.

• mvsRebuildIndex (optional, default=false)(LOCAL only):boolean value specifying whether the repository index should be rebuilt or not.

• mvsPersistentLocation (optional, default=none)(LOCAL only): an HDFS path where the MVSuggest directory will be saved.

• mvsIsOverwritePersistentLocation (optional, default=false): boolean argument that indicates whether an existing mvsPersistentLocation must be overwritten in case it already exists.

Example: Run a Categorization job to create a summary containing the records counts by continent, country, and state/provinces. The input is an existing spatial index called indexExample. The hierarchical data will be indexed and stored in HDFS at the path hierarchyIndex.

hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop/lib/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.Categorization indexName= indexExample  output=/user/hdfs/demo_vector/tweets/hier_count_spatial hierarchyInfo=vectoranalysis.categorization.WorldAdminHierarchyInfo hierarchyIndex=hierarchyIndex  hierarchyDataPaths=file:///templates/world_continents.json,file:///templates/world_countries.json,file:///templates/world_states_provinces.json spatialOperation=IsInside

Example: Run a Categorization job to create a summary of tweet counts per continent, country, states/provinces, and cities using MVSuggest.

hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop/lib/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.Categorization input="/user/hdfs/demo_vector/tweets/part*" inputFormat=<InputFormat_subclass> recordInfoProvider=<LocalizableRecordInfoProvider_subclass>  output=/user/hdfs/demo_vector/tweets/hier_count_mvs hierarchyInfo=vectoranalysis.categorization.WorldAdminHierarchyInfo hierarchyIndex=hierarchyIndex mvsLocation=file:///mvs_dir mvsMatchLayers=world_continents,world_countries,world_states_provinces spatialOperation=IsInside

To create a clustering result, use a command in the following format:

hadoop jar <HADOOP_LIB_PATH >/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.KMeansClustering [generic options] input=<path|comma_separated_paths|path_pattern> inputFormat=<InputFormat_subclass> recordInfoProvider=<RecordInfoProvider_subclass> output=<path> [srid=<integer_value>] [geodetic=<true|false>] [tolerance=<double_value>] [boundaries=<min_x,min_y,max_x,max_y>]  k=<number_of_clusters> [clustersPoints=<comma_separated_points_ordinates>] [deleteClusterFiles=<true|false>] [maxIterations=<integer_value>] [critFunClass=<CriterionFunction_subclass>] [shapeGenClass=<ClusterShapeGenerator_subclass>] [maxMemberDistance=<double_value>]

To use the new Hadoop API format, replace oracle.spatial.hadoop.vector.mapred.job.KMeansClustering with oracle.spatial.hadoop.vector.mapreduce.job.KMeansClustering.

Input/output arguments:

• input : the location of the input data. It can be expressed as a path, a comma separated list of paths, or a regular expression.

• inputFormat: the inputFormat class implementation used to read the input data.

• recordInfoProvider: the recordInfoProvider implementation used to extract information from the records read by the InputFormat class.

• output: the path where the spatial index will be stored

Spatial arguments:

• srid (optional, default=0): the spatial reference system (coordinate system) ID of the spatial data.

• geodetic (optional, default depends on the srid): Boolean value that indicates whether the geometries are geodetic or not.

• tolerance (optional, default=0.0): double value that represents the tolerance used when performing spatial operations.

• boundaries (optional, default=unbounded): the minimum and maximum values for each dimension, expressed as comma separated values in the form: minX,minY,maxX,maxY

• spatialOperation: the spatial operation to perform between the input data set and the hierarchical data set. Allowed values are IsInside and AnyInteract.

Clustering arguments:

• k: the number of clusters to be found.

• clusterPoints (optional, default=none): the initial cluster centers as a comma-separated list of point ordinates in the form: p1_x,p1_y,p2_x,p2_y,…,pk_x,pk_y

• deleteClusterFiles (optional, default=true): Boolean arguments that specifies whether the intermediate cluster files generated between iterations should be deleted or not

• maxIterations (optional, default=calculated based on the number k): the maximum number of iterations allowed before the job completes.

• critFunClass (optional, default=oracle.spatial.hadoop.vector.cluster.kmeans. SquaredErrorCriterionFunction) a fully qualified name of a CriterionFunction subclass.

• shapeGenClass (optional, default= oracle.spatial.hadoop.vector.cluster.kmeans. ConvexHullClusterShapeGenerator) a fully qualified name of a ClusterShapeGenerator subclass used to generate the geometry of the clusters.

• maxMemberDistance (optional, default=undefined): a double value that specifies the maximum distance between a cluster center and a cluster member.

Example: Run a Clustering job to generate 5 clusters. The generated clusters geometries will be the convex hull of all .

hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop/lib/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.KMeansClustering input="/user/hdfs/demo_vector/tweets/part*" output=/user/hdfs/demo_vector/tweets/result inputFormat=oracle.spatial.hadoop.vector.geojson.mapred.GeoJsonInputFormat recordInfoProvider=oracle.spatial.hadoop.vector.geojson.GeoJsonRecordInfoProvider srid=8307 geodetic=true tolerance=0.5 k=5 shapeGenClass=oracle.spatial.hadoop.vector.cluster.kmeans.ConvexHullClusterShapeGenerator

To create a binning result, use a command in the following format:

hadoop jar <HADOOP_LIB_PATH >/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.Binning [generic options] (indexName=<INDEX_NAME> [indexMetadataDir=<INDEX_METADATA_DIRECTORY>]) | (input=<DATA_PATH> inputFormat=<INPUT_FORMAT_CLASS> recordInfoProvider=<RECORD_INFO_PROVIDER_CLASS> [srid=<SRID>] [geodetic=<GEODETIC>] [tolerance=<TOLERANCE>]) output=<RESULT_PATH> cellSize=<CELL_SIZE> gridMbr=<GRID_MBR> [cellShape=<CELL_SHAPE>] [aggrFields=<EXTRA_FIELDS>]

To use the new Hadoop API format, replace oracle.spatial.hadoop.vector.mapred.job.Binning with oracle.spatial.hadoop.vector.mapreduce.job.Binning.

Input/output arguments:

• indexName: the name of an existing spatial index. The index information will be looked at the path given by indexMetadataDir. When used, the argument input is ignored.

• indexMetadataDir (optional, default=hdfs://server:port/user/<current_user>/oracle_spatial/index_metadata/): the directory where the spatial index metadata is located

• input : the location of the input data. It can be expressed as a path, a comma separated list of paths, or a regular expression.

• inputFormat: the inputFormat class implementation used to read the input data.

• recordInfoProvider: the recordInfoProvider implementation used to extract information from the records read by the InputFormat class.

• output: the path where the spatial index will be stored

Spatial arguments:

• srid (optional, default=0): the spatial reference system (coordinate system) ID of the spatial data.

• geodetic (optional, default depends on the srid): Boolean value that indicates whether the geometries are geodetic or not.

• tolerance (optional, default=0.0): double value that represents the tolerance used when performing spatial operations.

Binning arguments:

• cellSize: the size of the cells in the format: width,height

• gridMbr : the minimum and maximum dimension values for the grid in the form: minX,minY,maxX,maxY

• cellShape (optional, default=RECTANGLE): the shape of the cells. It can be RECTANGLE or HEXAGON

• aggrFields (optional, default=none): a comma-separated list of field names that will be aggregated.

Example: Run a spatial binning job to generate a grid of hexagonal cells and aggregate the value of the field SALES..

hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop/lib/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.Binning indexName=indexExample indexMetadataDir=indexMetadataDir output=/user/hdfs/demo_vector/result cellShape=HEXAGON cellSize=5 gridMbr=-175,-85,175,85 aggrFields=SALES

To perform spatial filtering, use a command in the following format:

hadoop jar <HADOOP_LIB_PATH >/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.SpatialFilter [generic options] ( indexName=<indexName> [indexMetadataDir=<path>] )  | 
( 
(input=<path|comma_separated_paths|path_pattern> isInputIndex=true)  | (input=<path|comma_separated_paths|path_pattern> inputFormat=<InputFormat_subclass> recordInfoProvider=<RecordInfoProvider_subclass>)
[srid=<integer_value>] [geodetic=<true|false>] [tolerance=<double_value>] [boundaries=<min_x,min_y,max_x,max_y>]
) output=<path>  spatialOperation=<[IsInside, AnyInteract]> queryWindow=<json-geometry>

To use the new Hadoop API format, replace oracle.spatial.hadoop.vector.mapred.job.SpatialFilter with oracle.spatial.hadoop.vector.mapreduce.job.SpatialFilter.

Input/output arguments:

• indexName: the name of an existing spatial index. The index information will be looked at the path given by indexMetadataDir. When used, the argument input is ignored.

• indexMetadataDir (optional, default=hdfs://server:port/user/<current_user>/oracle_spatial/index_metadata/): the directory where the spatial index metadata is located

• input : the location of the input data. It can be expressed as a path, a comma separated list of paths, or a regular expression.

• inputFormat: the inputFormat class implementation used to read the input data.

• recordInfoProvider: the recordInfoProvider implementation used to extract information from the records read by the InputFormat class.

• output: the path where the spatial index will be stored

Spatial arguments:

• srid (optional, default=0): the spatial reference system (coordinate system) ID of the spatial data.

• geodetic (optional, default depends on the srid): Boolean value that indicates whether the geometries are geodetic or not.

• tolerance (optional, default=0.0): double value that represents the tolerance used when performing spatial operations.

Binning arguments:

• cellSize: the size of the cells in the format: width,height

• gridMbr : the minimum and maximum dimension values for the grid in the form: minX,minY,maxX,maxY

• cellShape (optional, default=RECTANGLE): the shape of the cells. It can be RECTANGLE or HEXAGON

• aggrFields (optional, default=none): a comma-separated list of field names that will be aggregated.

• boundaries (optional, default=unbounded): the minimum and maximum values for each dimension, expressed as comma separated values in the form: minx,minY,maxX,maxY

• spatialOperation: the operation to be applied between the queryWindow and the geometries from the input data set

• queryWindow: the geometry used to filter the input dataset.

Example: Perform a spatial filtering operation.

hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop/lib/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.SpatialFilter indexName=indexExample indexMetadataDir=indexMetadataDir output=/user/hdfs/demo_vector/result spatialOperation=IsInside queryWindow='{"type":"Polygon", "coordinates":[[-106, 25, -106, 30, -104, 30, -104, 25, -106, 25]]}'

To create a job to get location suggestions, use a command in the following format.

hadoop jar <HADOOP_LIB_PATH >/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.SuggestService [generic options] input=<path|comma_separated_paths|path_pattern> inputFormat=<InputFormat_subclass> recordInfoProvider=<RecordInfoProvider_subclass> output=<path> mvsLocation=<path|URL> [mvsMatchLayers=<comma_separated_layers>] [mvsMatchCountry=<country_name>] [mvsSpatialResponse=<[NONE, FEATURE_GEOMETRY, FEATURE_CENTROID]>] [mvsInterfaceType=<[UNDEFINED, LOCAL, WEB]>] [mvsIsRepository=<true|false>] [mvsRebuildIndex=<true|false>] [mvsPersistentLocation=<hdfs_path>] [mvsOverwritePersistentLocation=<true|false>] [mvsMaxRequestRecords=<integer_number>]

To use the new Hadoop API format, replace oracle.spatial.hadoop.vector.mapred.job.SuggestService with oracle.spatial.hadoop.vector.mapreduce.job.SuggestService.

Input/output arguments:

• input : the location of the input data. It can be expressed as a path, a comma separated list of paths, or a regular expression. (Ignored if indexName is specified.)

• inputFormat: the inputFormat class implementation used to read the input data. (Ignored if indexName is specified.)

• recordInfoProvider: the recordInfoProvider implementation used to extract information from the records read by the InputFormat class. (Ignored if indexName is specified.)

• output: the path where the spatial index will be stored

MVSuggest arguments:

• mvsLocation: The path to the MVSuggest directory or repository for local standalone instances of MVSuggest or the service URL when working with a remote instance. This argument is required when working with MVSuggest.

• mvsMatchLayers (optional, default=all): comma separated list of layers. When provided, MVSuggest will only use these layers to perform the search.

• mvsMatchCountry (optional, default=none): a country name which MVSuggest will give higher priority when performing matches.

• mvsSpatialResponse (optional, default=CENTROID): the type of the spatial results contained in each returned match. It can be one of the following values: NONE, FEATURE_GEOMETRY, FEATURE_CENTROID.

• mvsInterfaceType (optional: default=LOCAL): the type of MVSuggest service used, it can be LOCAL or WEB.

• mvsIsRepository (optional: default=false) (LOCAL only): Boolean value that specifies whether mvsLocation points to a whole MVS directory(false) or only to a repository(true). An MVS repository contains only JSON templates; it may or not contain a _config_ and _geonames_ folder.

• mvsRebuildIndex (optional, default=false)(LOCAL only):boolean value specifying whether the repository index should be rebuilt or not.

• mvsPersistentLocation (optional, default=none)(LOCAL only): an HDFS path where the MVSuggest directory will be saved.

• mvsIsOverwritePersistentLocation (optional, default=false): boolean argument that indicates whether an existing mvsPersistentLocation must be overwritten in case it already exists.

Example: Get suggestions based on location texts from the input data set.

hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop/lib/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.SuggestService input="/user/hdfs/demo_vector/tweets/part*" inputFormat=<InputFormat_subclass> recordInfoProvider=<LocalizableRecordInfoProvider_subclass> output=/user/hdfs/demo_vector/tweets/suggest_res mvsLocation=file:///mvs_dir mvsMatchLayers=world_continents,world_countries,world_states_provinces

To perform a spatial join operation on two data sets, use a command in the following format.

hadoop jar <HADOOP_LIB_PATH >/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job. SpatialJoin [generic options] 
inputList={ 
 { 
  ( indexName=<dataset1_spatial_index_name>  indexMetadataDir=<dataset1_spatial_index_metadata_dir_path> ) 
  |  
  ( input=<dataset1_path|comma_separated_paths|path_pattern> inputFormat=<dataset1_InputFormat_subclass> recordInfoProvider=<dataset1_RecordInfoProvider_subclass> )  
  [boundaries=<min_x,min_y,max_x,max_y>]  
 }  
 { 
  (indexName=<dataset2_spatial_index_name> indexMetadataDir=<dataset2_spatial_index_metadata_dir_path> 
  ) 
  |  
  ( input=<dataset2_path|comma_separated_paths|path_pattern> inputFormat=<dataset2_InputFormat_subclass> recordInfoProvider=<dataset2_RecordInfoProvider_subclass> 
  )  
  [boundaries=<min_x,min_y,max_x,max_y>]  
 } 
} output=<path>[srid=<integer_value>] [geodetic=<true|false>] [tolerance=<double_value>] boundaries=<min_x,min_y,max_x,max_y> spatialOperation=<AnyInteract|IsInside|WithinDistance> [distance=<double_value>] [samplingRatio=<decimal_value_between_0_and_1> | partitioningResult=<path>]

To use the new Hadoop API format, replace oracle.spatial.hadoop.vector.mapred.job.SpatialJoin with oracle.spatial.hadoop.vector.mapreduce.job.SpatialJoin.

InputList: A list of two input data sets. The list is enclosed by curly braces ({}). Each list element is an input data set, which is enclosed by curly braces. An input data set can contain the following information, depending on whether the data set is specified as a spatial index.

If specified as a spatial index:

• indexName: the name of an existing spatial index.

• indexMetadataDir : the directory where the spatial index metadata is located

If not specified as a spatial index:

• input : the location of the input data. It can be expressed as a path, a comma separated list of paths, or a regular expression. (Ignored if indexName is specified.)

• inputFormat: the inputFormat class implementation used to read the input data. (Ignored if indexName is specified.)

• recordInfoProvider: the recordInfoProvider implementation used to extract information from the records read by the InputFormat class. (Ignored if indexName is specified.)

output: the path where the results will be stored

Spatial arguments:

• srid (optional, default=0): the spatial reference system (coordinate system) ID of the spatial data.

• geodetic (optional, default depends on the srid): boolean value that indicates whether the geometries are geodetic or not.

• tolerance (optional, default=0.0): double value that represents the tolerance used when performing spatial operations.

• boundaries (optional, default=unbounded): the minimum and maximum values for each dimension, expressed as comma separated values in the form: minX,minY,maxX,maxY

• spatialOperation: the spatial operation to perform between the input data set and the hierarchical data set. Allowed values are IsInside and AnyInteract.

• distance: distance used for WithinDistance operations.

Partitioning arguments:

• samplingRatio (optional, default=0.1): ratio used to sample the data sets when partitioning needs to be performed

• partitioningResult (optional, default=none): Path to a previously generated partitioning result file

Example: Perform a spatial join on two data sets.

hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop/lib/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.SpatialJoin inputList="{{input=/user/hdfs/demo_vector/world_countries.json inputFormat=oracle.spatial.hadoop.vector.geojson.mapred.GeoJsonInputFormat recordInfoProvider=oracle.spatial.hadoop.vector.geojson.GeoJsonRecordInfoProvider} {input=file="/user/hdfs/demo_vector/tweets/part*” inputFormat=oracle.spatial.hadoop.vector.geojson.mapred.GeoJsonInputFormat recordInfoProvider=oracle.spatial.hadoop.vector.geojson.GeoJsonRecordInfoProvider}}" output=/user/hdfs/demo_vector/spatial_join srid=8307 spatialOperation=AnyInteract boundaries=-180,-90,180,90

To perform a spatial partitioning, use a command in the following format.

hadoop jar <HADOOP_LIB_PATH >/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job. SpatialJoin [generic options] 
inputList={ 
 { 
  ( indexName=<dataset1_spatial_index_name>  indexMetadataDir=<dataset1_spatial_index_metadata_dir_path> ) 
  |  
  ( input=<dataset1_path|comma_separated_paths|path_pattern> inputFormat=<dataset1_InputFormat_subclass> recordInfoProvider=<dataset1_RecordInfoProvider_subclass> )  
  [boundaries=<min_x,min_y,max_x,max_y>]  
 }
[  
 { 
  (indexName=<dataset2_spatial_index_name> indexMetadataDir=<dataset2_spatial_index_metadata_dir_path> 
  ) 
  |  
  ( input=<dataset2_path|comma_separated_paths|path_pattern> inputFormat=<dataset2_InputFormat_subclass> recordInfoProvider=<dataset2_RecordInfoProvider_subclass> 
  )  
  [boundaries=<min_x,min_y,max_x,max_y>]  
 }
  ……
 { 
  (indexName=<datasetN_spatial_index_name> indexMetadataDir=<datasetN_spatial_index_metadata_dir_path> 
  ) 
  |  
  ( input=<datasetN_path|comma_separated_paths|path_pattern> inputFormat=<datasetN_InputFormat_subclass> recordInfoProvider=<datasetN_RecordInfoProvider_subclass> 
  )  
  [boundaries=<min_x,min_y,max_x,max_y>]  
 }
 
}
] output=<path>[srid=<integer_value>] [geodetic=<true|false>] [tolerance=<double_value>] boundaries=<min_x,min_y,max_x,max_y> [samplingRatio=<decimal_value_between_0_and_1>]

To use the new Hadoop API format, replace oracle.spatial.hadoop.vector.mapred.job.Partitioning with oracle.spatial.hadoop.vector.mapreduce.job.Partitioning.

InputList: A list of two input data sets. The list is enclosed by curly braces ({}). Each list element is an input data set, which is enclosed by curly braces. An input data set can contain the following information, depending on whether the data set is specified as a spatial index.

If specified as a spatial index:

• indexName: the name of an existing spatial index.

• indexMetadataDir : the directory where the spatial index metadata is located

If not specified as a spatial index:

• input : the location of the input data. It can be expressed as a path, a comma separated list of paths, or a regular expression. (Ignored if indexName is specified.)

• inputFormat: the inputFormat class implementation used to read the input data. (Ignored if indexName is specified.)

• recordInfoProvider: the recordInfoProvider implementation used to extract information from the records read by the InputFormat class. (Ignored if indexName is specified.)

output: the path where the results will be stored

Spatial arguments:

• srid (optional, default=0): the spatial reference system (coordinate system) ID of the spatial data.

• geodetic (optional, default depends on the srid): boolean value that indicates whether the geometries are geodetic or not.

• tolerance (optional, default=0.0): double value that represents the tolerance used when performing spatial operations.

• boundaries (optional, default=unbounded): the minimum and maximum values for each dimension, expressed as comma separated values in the form: minX,minY,maxX,maxY

Partitioning arguments:

• samplingRatio (optional, default=0.1): ratio used to sample the data sets when partitioning needs to be performed

Example: Partition two data sets.

hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop/lib/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.Partitioning inputList="{{input=/user/hdfs/demo_vector/world_countries.json inputFormat=oracle.spatial.hadoop.vector.geojson.mapred.GeoJsonInputFormat recordInfoProvider=oracle.spatial.hadoop.vector.geojson.GeoJsonRecordInfoProvider} {input=file="/user/hdfs/demo_vector/tweets/part*” inputFormat=oracle.spatial.hadoop.vector.geojson.mapred.GeoJsonInputFormat recordInfoProvider=oracle.spatial.hadoop.vector.geojson.GeoJsonRecordInfoProvider}}" output=/user/hdfs/demo_vector/partitioning srid=8307 boundaries=-180,-90,180,90

Multiple input data sets can be specified to a Vector job through the command line interface using the inputList parameter. The inputList parameter value is a group of input data sets. The inputList parameter format is as follows:

inputList={ {input_data_set_1_params} {input_data_set_2_params} … {input_data_set_N_params} }

Each individual input data set can be one of the following input data sets:

• Non-file input data set: inputFormat=<InputFormat_subclass> recordInfoProvider=<RecordInfoProvider_subclass> [srid=<integer_value>] [geodetic=<true|false>] [tolerance=<double_value>] [boundaries=<min_x,min_y,max_x,max_y>]

• File input data set: input=<path|comma_separated_paths|path_pattern> inputFormat=<FileInputFormat_subclass> recordInfoProvider=<RecordInfoProvider_subclass> [srid=<integer_value>] [geodetic=<true|false>] [tolerance=<double_value>] [boundaries=<min_x,min_y,max_x,max_y>]

• Spatial index input data set: ( ( indexName=<<indexName>> [indexMetadataDir=<<path>>]) | ( isInputIndex=<true> input=<path|comma_separated_paths|path_pattern> ) ) [srid=<integer_value>] [geodetic=<true|false>] [tolerance=<double_value>] [boundaries=<min_x,min_y,max_x,max_y>]

• NoSQL input data set: kvStore=<kv store name> kvStoreHosts=<comma separated list of hosts> kvParentKey=<parent key> [kvConsistency=<Absolute|NoneRequired|NoneRequiredNoMaster>] [kvBatchSize=<integer value>] [kvDepth=<CHILDREN_ONLY|DESCENDANTS_ONLY|PARENT_AND_CHILDREN|PARENT_AND_DESCENDANTS>] [kvFormatterClass=<fully qualified class name>] [kvSecurity=<properties file path>] [kvTimeOut=<long value>] [kvDefaultEntryProcessor=<fully qualified class name>] [kvEntryGrouper=<fully qualified class name>] [ kvResultEntries={ { minor key 1: a minor key name relative to the major key [fully qualified class name: a subclass of NoSQLEntryProcessor class used to process the entry with the given key] } * } ] [srid=<integer_value>] [geodetic=<true|false>] [tolerance=<double_value>] [boundaries=<min_x,min_y,max_x,max_y>]

Notes:

• A Categorization job does not support multiple input data sets.

• A SpatialJoin job only supports two input data sets.

• A SpatialIndexing job does not accept input data sets of type spatial index.

• NoSQL input data sets can only be used when kvstore.jar is present in the classpath.

1. Open the console: http://<oracle_big_data_spatial_vector_console>:8045.
2. Click the Raster tab.
3. Click Select File or Path and browse to the demo folder that contains a set of Hawaii images (/opt/shareddir/spatial/data/rasters).
4. By default, Spark is selected. If you want to use Hadoop, click the Use Spark button to change it to Use Hadoop.
5. Select the rasters folder and click Load images.

   You will receive a message about the job being accepted, with a tracking URL. You can track the job status using that URL.

   After the job finishes, you can see the uploaded images in the globe in the Viewer tab.

1. Open the console: http://<oracle_big_data_spatial_vector_console>:8045.
2. Click the Raster tab.
3. Click the Hadoop Viewer tab.
4. Click Refresh Footprints to update the footprints in the globe, and wait until all footprints are displayed on the globe.

   Identical rasters are displayed with a yellow edge

1. Right click over a raster. If you select a raster with a red or yellow edge, a tooltip with a list of rasters may appear.
2. Click Process Rasters with Same MBR. You can exclude rasters from the process by clicking the X button on the left side of every row. If single raster was select, click Process Image (No Mosaic).
   The Raster Process dialog box is displayed.
3. By default, Spark is selected to process the job. To use Hadoop instead, click Use Spark to toggle the button to Use Hadoop.
4. In the Raster Process dialog, scroll down and click Create Mosaic.
   Wait until the raster processing is finished. The result will displayed in the Result tab.
5. Optionally, download the result by clicking Download Full Size Image below the result image.

1. Open the console: http://<oracle_big_data_spatial_vector_console>:8045.
2. Click the Raster tab.
3. Click the Hadoop Viewer tab.
4. Click Refresh Footprints to update the footprints in the globe, and wait until all footprints are displayed on the globe.

   Identical rasters are displayed with a yellow edge

5. Click Select and crop coordinates of Footprints.
6. Draw a rectangle that wraps the rasters (at least one) and desired area, zooming in or out as necessary.
7. Right-click on the map and select Generate Mosaic.
   The raster process dialog is displayed.
8. By default, Spark is select to process the job. To use Hadoop instead, click Use Spark to toggle the button to Use Hadoop.
9. In the raster process dialog, scroll down and click Create Mosaic.

   Wait until the raster processing is finished. The result will displayed in the Result tab.

10. Optionally, download the result by clicking Download Full Size Image.

1. Click Advanced options.

   A group of new elements is displayed for adding add the advanced options.

2. Scroll down until you see the raster operations.
3. Choose a raster operation from the list. If you want to add a complex operation, toggle the Hide Complex Operations checkbox.

   Only one complex operation is allowed per raster processing action.

4. After you select an operation from the list on the left, add it to the process by clicking the right arrow.

   Some operations also require parameters.

5. Add more operations if you want.

   To remove an operation, select it in the list on the right and click the left arrow. You can also remove all operations in the list.

6. By default, Spark is selected to process the job. To use Hadoop instead, click Use Spark to toggle the button to Use Hadoop.
7. Click Create Mosaic.

   Wait until the raster processing is finished. The result will displayed in the Result tab.

8. Optionally, download the result by clicking Download Full Size Image.

1. Open the console: http://<oracle_big_data_spatial_vector_console>:8045.
2. Click the Raster tab.
3. Click the Hadoop Viewer tab.
4. Click Refresh Footprints to update the footprints in the globe, and wait until all footprints are displayed on the globe.

   Identical rasters are displayed with a yellow edge

5. Click Select and crop coordinates of Footprints.
6. Draw a rectangle that wraps the rasters (at least one) and desired area, zooming in or out as necessary.
7. Right-click on the map and select Generate Mosaic.
   The raster process dialog is displayed.
8. By default, Spark is select to process the job. To use Hadoop instead, click Use Spark to toggle the button to Use Hadoop.
9. Select the appropriate Pixel Type
   Usually these images are Float 32 Bits.
10. Click Advanced Options.
    You will see a group of new elements to add as advanced options.
11. Scroll down until you see the Process Classes controls.
12. Specify the Fully Qualified Class Name, then click Add.
    The framework provides a default process class for slope: oracle.spatial.hadoop.imageprocessor.process.ImageSlope
13. Click Create Mosaic
    Wait until the raster processing is finished.
    The result will be displayed in the Result tab.

1. Select the desired image Output Format.
2. By default, Spark is select to process the job. To use Hadoop instead, click Use Spark to toggle the button to Use Hadoop.
3. Scroll down and click Create Mosaic.

   Wait until the raster processing is finished. The result will be displayed in the Result tab.

4. Optionally, download the result by clicking Download Full Size Image.