Fraska Portal

Exploring the WebSphere Commerce world

WebSphere Commerce, Data Load and SOLR Delta Index

Posted by on in Data Load
  • Font size: Larger Smaller
  • Hits: 13356
  • 121 Comments
  • Subscribe to this entry
  • Print

One of the most annoying thing with the previous version of WCS (until de v7 FEP5) was the impossibility to perform a delta build of SOLR index after a data load process also for few line changes  ... basically a wast of time and resources.

However, FEP5 has finally introduced the possibility to make it works. In fact, new mediators have been implemented to load the data load changes in the temporary index tables and than execute successfully a Delta build reindex.

To test this interesting and very useful feature I realized a specific use case with Aurora on Windows platform, WCS Toolkit Enterprise v7 FEP6, DB2 and SOLR.

The test case:

  1. Create a simple CSV file with the introduction of a new product: "Fancy Laptop"
  2. Configure the data load (not using the new mediators) to load this product
  3. Run the data load
    • test: the product does not appear in the Front End
  4. Run a DELTA rebuild of the SOLR index
    • test: the product does not appear in the Front End
  5. Run a FULL rebuild of the SOLR index
    • test: the product appears in the Front End
  6. Update data load configuration (adding the use of the new mediators and properties)
  7. Change the product's name in the CSV ("New Fancy Laptop")
  8. Run the data load
  9. Run a DELTA rebuild of the SOLR index
    • test: the product appears in the Front End

 1. CSV content

The following contains the content of the CSV file used to load the new product "Fancy Laptop".

 
CatalogEntry,,,,,,,,,,,,,
PartNumber,Type,ParentPartNumber,Sequence,ParentGroupIdentifier,CurrencyCode,ListPrice,Price,Name,ShortDescription,LongDescription,Thumbnail,FullImage,Delete,Buyable,Manufacturer
CLA022_2210,Product,,1,Computers Laptops,USD,600,585,Fancy Laptop,A fancy and colorful laptop.,This fancy laptop is lightweight and can handle everyday tasks like reading stories and watching videos.,images/catalog/electronics/cla022_laptops/70x70/cla022_2210.jpg,images/catalog/electronics/cla022_laptops/447x447/cla022_2210.jpg,0,1,DVR Technics

 

2. Data Load configuration to load this new product with name, generic info and price

In order to load the new product, using the data load, we can use a configuration allows to map the csv reported above. Follow an extract of the configuration:

<_config:LoadOrder commitCount="100" batchSize="1" dataLoadMode="Replace" >
<_config:property name="firstTwoLinesAreHeader" value="true" />
       
<_config:LoadItem name="Products" businessObjectConfigFile="Products/wc-loader-product.xml" >
<_config:DataSourceLocation location="Products/Products_Aurora.csv" />
        
       
<_config:LoadItem name="Prices" businessObjectConfigFile="Prices/wc-loader-offer.xml" >
<_config:DataSourceLocation location="Prices/Prices_Aurora.csv" />
        
  
    

 

3.  Run the data load

Using the dataload script and referring the above configuration and CSV file we can load the new product. Once data load is over (just few seconds... it's 1 line) the Management Center will show the new product up.

 WebSphere Commerce Catalog Management Center

However, since we did not run any SOLR index rebuild the product will not show up in the front end.

 

4.  Run DELTA index rebuild

Since we did not update the Data Load configuration with the last mediators provided with FEP5, running a delta rebuild will not udpate the indexes and we could not find the product in the Front End.

The product changes affecting the Delta are temporary stored in the table TI_DELTA_CATENTRY. In fact the following query does not return anything related to the new product:

SELECT * FROM TI_DELTA_CATENTRY;

 

5.  Run FULL index rebuild

We can get the change in the front end just making a full rebuild of the index, much more expensive in terms of time and resources but at least it allows to have the index updated with the last changes.

WebSphere Commerce Front End 

 do you see the "Fancy Laptop"?

 

6.  Update the data load configuration to avoid FULL index rebuild

In order to avoid the running of a full rebuild every time we load piece of data with DL, we should use the following configuration:

<_config:LoadOrder commitCount="100" batchSize="1" dataLoadMode="Replace" >
<_config:property name="firstTwoLinesAreHeader" value="true" />
<_config:property name="buildSearchIndex" value="delta" />
<_config:property name="DeltaIndexingThresholdUponWorkspaceApproval" value="100000" />

       
<_config:LoadItem name="Products" businessObjectConfigFile="Products/wc-loader-product.xml" >
<_config:DataSourceLocation location="Products/Products_Aurora.csv" />
        
       
<_config:LoadItem name="Delta Products" businessObjectConfigFile="Products/wc-loader-product-for-delta-index.xml" >
<_config:DataSourceLocation location="Products/Products_Aurora.csv" />
        
       
             
<_config:LoadItem name="Prices" businessObjectConfigFile="Prices/wc-loader-offer.xml" >
<_config:DataSourceLocation location="Prices/Prices_Aurora.csv" />
        
       
           
    

Please note, the buildSearchIndex property allows to start automatically- just after a successful run of the data load- the delta build of the index. In case you are using the scheduled job UpdateSearchIndex or you need to start the build manually (like in this example), just remove this property.

wc-loader-product-for-delta-index.xml is the mapping file using the new Mediator: CatalogEntrySearchIndexMediator. Follow an extraction of the content of this configuration file:

...

<_config:BusinessObjectMediator className="com.ibm.commerce.catalog.dataload.mediator.CatalogEntrySearchIndexMediator" componentId="com.ibm.commerce.catalog" >

...
      

 

7. CSV file with a different product name

To test the new configuration we can try to load a different name for the new product, for example "Fancy Laptop V2". Just matter of update the CSV in the field related to the name.

 

8. Run the data load 

This time we can realize the data load will also load data in the table TI_DELTA_CATENTRY, that's a good clue to understand delta will work.

 

9. Run DELTA index rebuild

After a successful running of the delta rebuild process we will find the product in the Front End, without the need to run a full, saving time and maybe money :-) 

 

You can download the Data Load working configuration of above example from here.

Info center reference: Catalog Upload operations in WebSphere Commerce search

Rate this blog entry:
0

Comments

Leave your comment

Guest
Guest Wednesday, 17 October 2018

Most Popular Post

WebSphere Commerce, the SOLR extension index
Administration
Rate this blog entry:
4
WebSphere Commerce, Data Load and SOLR Delta Index
Data Load
Rate this blog entry:
0
WebSphere Commerce, the curious life of a front-end catalog request
Core
Rate this blog entry:
5

Latest Blogs

WebSphere Commerce, CommandLevelAuthorizationCache
Cache
Rate this blog entry:
0
WebSphere Commerce v8, toolkit exception, ClassNotFound db2
Administration
Rate this blog entry:
1
WebSphere Commerce, ATP migration
Store
Rate this blog entry:
0
WebSphere Commerce, the curious life of a front-end catalog request
Core
Rate this blog entry:
5
WebSphere Commerce, Performance analysis of few European stores
Performance
Rate this blog entry:
0