Fraska Portal

Exploring the WebSphere Commerce world

Subscribe to this list via RSS Blog posts tagged in Index
Posted by on in Administration
WebSphere Commerce, the SOLR extension index

One of the things I like most about the last enhancements- maybe introduced with FEP6, not sure of that- is the possibility to create an extension of the SOLR index. An extension index is, in a nutshell, an index which complements the data provided by the base index. A enlightening explanation is reported in the following IC page: Extending the WebSphere Commerce search base index schema.

According to the Info Center: "... dynamic information such as product inventory or ratings change more frequently. Therefore, it is put into a separate index, where it can be refreshed at different intervals than the base product index". That's awesome and it looks exactly the basic idea under the concept of Inventory Index (introduced with FEP6).

b2ap3_thumbnail_IC_SOLR_extension_index.png

Above is an extraction of the image reported in the IC page already mentioned before.

Using this feature we can finally think to implement re-index strategies with high frequency, pretty useful for dynamic data. The basic bricks used by the WCS team to build up this new feature are- according to my investigation- a custom Search Component (SolrSearchMultipleQueryComponent) and the possibility to make Join between different indexes.

That combination gives to the WCS implementation of SOLR an added value makes the SOLR native Join feature transparent to the developers. Let's review the main things ...

WebSphere Commerce, FEP 7, SOLR index pre-process error

Once the FEP7 installation is done and the starter store is published with some example data, the next step is the SOLR index configuration (setupSearchIndex), pre-process (di-preprocess) and build (di-buildindex).

However, during the pre-process step I faced an interesting exception. In fact, running the following command

di-preprocess.bat <WC_installfolder>\search\pre-processConfig\MC_<catalogId>\DB2

the process ended with the exception:

SEVERE: Exception message: CWFDIH0001: Data Import encountered a generic error. The error is null. , stack trace: com.ibm.commerce.foundation.dataimport.exception.DataImportSystemException: CWFDIH0001: Data Import encountered a generic error. The error is null. 
	at com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain.processDataConfig(DataImportPreProcessorMain.java:957)
	at com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain.execute(DataImportPreProcessorMain.java:802)
	at com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain.main(DataImportPreProcessorMain.java:289)
Caused by: java.lang.NullPointerException
	at com.ibm.commerce.foundation.dataimport.preprocess.StaticAttributeDataPreProcessor.process(StaticAttributeDataPreProcessor.java:299)
	at com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain.processDataConfig(DataImportPreProcessorMain.java:919)
	... 2 more
WebSphere Commerce, the "dark side" of the interactions between SOLR delta index and the DataLoad

This topic's title could seems a bit alarming but it's just a pinch of irony I would give, most of all because I struggled for so many weeks on this feature that it was like fighting against the "dark". However, I just want to focus on few things we should think about before to plan the introduction of the delta index when the environment is updated through the data load.

In particular, if we have to introduce the delta index- in place of the full- and the data load is already set up, or we have to plan from scratch the data load process, we should keep in mind that the data load operations, the order they are executed and the delta updates recorded in the temporary delta tables (TI_DELTA_CATENTRY and TI_DELTA_CATGROUP) are strictly connected; we should not consider the data load a black box respect to the SOLR index, when we need to update the system through the delta mode. The SOLR index and the Data Load are unfortunately tied in a way we cannot think to build a loading process without knowing in which way they interact.

Ok, let's leave fancy words! I will report two specific cases: 

  1. delete bundles;
  2. delete sales categories (second part of this article);
WebSphere Commerce, Data Load and SOLR Delta Index

One of the most annoying thing with the previous version of WCS (until de v7 FEP5) was the impossibility to perform a delta build of SOLR index after a data load process also for few line changes  ... basically a wast of time and resources.

However, FEP5 has finally introduced the possibility to make it works. In fact, new mediators have been implemented to load the data load changes in the temporary index tables and than execute successfully a Delta build reindex.

To test this interesting and very useful feature I realized a specific use case with Aurora on Windows platform, WCS Toolkit Enterprise v7 FEP6, DB2 and SOLR.

The test case:

  1. Create a simple CSV file with the introduction of a new product: "Fancy Laptop"
  2. Configure the data load (not using the new mediators) to load this product
  3. Run the data load
    • test: the product does not appear in the Front End
  4. Run a DELTA rebuild of the SOLR index
    • test: the product does not appear in the Front End
  5. Run a FULL rebuild of the SOLR index
    • test: the product appears in the Front End
  6. Update data load configuration (adding the use of the new mediators and properties)
  7. Change the product's name in the CSV ("New Fancy Laptop")
  8. Run the data load
  9. Run a DELTA rebuild of the SOLR index
    • test: the product appears in the Front End

Most Popular Post

WebSphere Commerce, the SOLR extension index
Administration
Rate this blog entry:
4
WebSphere Commerce, Data Load and SOLR Delta Index
Data Load
Rate this blog entry:
0
WebSphere Commerce, the curious life of a front-end catalog request
Core
Rate this blog entry:
5

Latest Blogs

WebSphere Commerce, CommandLevelAuthorizationCache
Cache
Rate this blog entry:
0
WebSphere Commerce v8, toolkit exception, ClassNotFound db2
Administration
Rate this blog entry:
1
WebSphere Commerce, ATP migration
Store
Rate this blog entry:
0
WebSphere Commerce, the curious life of a front-end catalog request
Core
Rate this blog entry:
5
WebSphere Commerce, Performance analysis of few European stores
Performance
Rate this blog entry:
0