Fraska Portal

Exploring the WebSphere Commerce world

WebSphere Commerce, the "dark side" of the interactions between SOLR delta index and the DataLoad

Posted by on in Data Load
  • Font size: Larger Smaller
  • Hits: 9134
  • 30 Comments
  • Subscribe to this entry
  • Print
WebSphere Commerce, the "dark side" of the interactions between SOLR delta index and the DataLoad

This topic's title could seems a bit alarming but it's just a pinch of irony I would give, most of all because I struggled for so many weeks on this feature that it was like fighting against the "dark". However, I just want to focus on few things we should think about before to plan the introduction of the delta index when the environment is updated through the data load.

In particular, if we have to introduce the delta index- in place of the full- and the data load is already set up, or we have to plan from scratch the data load process, we should keep in mind that the data load operations, the order they are executed and the delta updates recorded in the temporary delta tables (TI_DELTA_CATENTRY and TI_DELTA_CATGROUP) are strictly connected; we should not consider the data load a black box respect to the SOLR index, when we need to update the system through the delta mode. The SOLR index and the Data Load are unfortunately tied in a way we cannot think to build a loading process without knowing in which way they interact.

Ok, let's leave fancy words! I will report two specific cases: 

  1. delete bundles;
  2. delete sales categories (second part of this article);

Delete bundles

Delete bundles using Management Center

Stealing the Info Center definition, "A bundle is a collection of catalog entries that allow customers to buy multiple items with one click", so in other words, a bundle is a simple catentry with one or more components, where each component is another catentry. 

The management center allows to create a bundle using the "Catalog tool" option "Create a bundle", filling all the needed fields and finally add the components.

 b2ap3_thumbnail_SOLR_1.png

Of course, it allows to delete a bundle, simply selecting the item's menu "Delete"

b2ap3_thumbnail_SOLR_2.png

Ok, till now nothing really new. They are basic operations in CMC. However, it's useful understand what happens under the cover when a delete operation is done. As soon as we delete a bundle, the temporary delta table TI_DELTA_CATENTRY will be populated properly. It will contain:

  • a row indicating the 'D'- delete operation- of the bundle;
  • a row for each component (of the deleted bundle) indicating the 'U'- update operation- of the component.

So for example, let's suppose I delete with CMC the bundle 221701 which has one component: 141047.

The TI_DELTA_CATENTRY will contain the following data:

MASTERCATALOG_ID;CATENTRY_ID;ACTION;LASTUPDATE
10001;141047;U;2013-11-25 17:28:14.994431
10001;221701;D;2013-11-25 17:28:14.996485  

Running the UpdateSearchIndex (in mode = 1, delta mode) the bundle will properly disappear from the front end, meaning the delta updates is working fine.

So far so good, since we are using the management center. The troubles could come if we use the data load to delete the bundles, a really common scenario!

Delete bundles using the Data Load

If we need to remove bundles we can follow the Info Center indications (keep your attention on "Delete" paragraph):

http://pic.dhe.ibm.com/infocenter/wchelp/v7r0m0/topic/com.ibm.commerce.data.doc/refs/rmlinsertbundlekit.htm

Keeping in mind we also need to update the delta tables, to have a working data load, we need:

  • an environment XML configuration file;
  • the main XML configuration file;
  • a loader XML configuration file uses two different mediators: CatalogEntryIndexMediator (to load the updates in the catalog) CatalogEntrySearchIndexMediator (to load the updates in the delta tables);
  • a CSV file like the one reported in the Info Center.

You can find the sample files IBM provides in the Data Load sample folder. In particular, we should focus on the loader file; I attach an example (the IBM's one): wc-loader-catalog-entry.xml.

You can also find it in your environment under the folder: WC/xml/config/com.ibm.commerce.catalog/dataload/.

Once again, our aim is to delete the bundle; running the data load, in particular the Info Center example reported just above, we will see the TI_DELTA_CATENTRY got populated but in a different way than the Management Center does. In fact, this time it will contain just the following data:

MASTERCATALOG_ID;CATENTRY_ID;ACTION;LASTUPDATE
10001; 219201;D;2013-11-25 17:28:14.996485 

You see? It's missing the "Update" action of the components. 

The problem

Despite the data load has followed our indication and deleted the bundle (in the DB the requested bundle will be marked as "mark for delete") and the UpdateSearchIndex scheduled job ran in delta mode without any apparent issue, the bundle is still visible in the front end cause it's still in the SOLR index indeed, like if nothing happened!

Basically, that happens cause the mediator CatalogEntrySearchIndexMediator does not add the "update" actions of the components. It had to work exactly like the Management Center but it does not.

The solution 

Once we understood they way the search mediators work, we know how we have to set up the dataload. In this specific case, the datalod was made by just a single step:

  1. delete the bundle

To overcome the mentioned problem, we need to add a further step in the loading process. In particular:

  1. delete the bundles components
  2. delete the bundle

It means, we will need:

  • an environment XML configuration file;
  • the main XML configuration file;
  • a loader XML configuration file- to map the bundles components- uses two different mediators: CatalogEntryIndexMediator (to load the updates in the catalog) CatalogEntrySearchIndexMediator (to load the updates in the delta tables);
  • a loader XML configuration file- to map the bundles- uses two different mediators: CatalogEntryIndexMediator (to load the updates in the catalog) CatalogEntrySearchIndexMediator (to load the updates in the delta tables);
  • a CSV file like the one reported in the Info Center.

In this way the data load will process two steps: delete of bundles components and than delete of bundles. The temporary index delta table will be populated exactly as the Management center does, and the SOLR index will be updated properly.

Soon I'll provide further info about this interactions between SOLR delta index and Data Load, in particular discussing about the operation of delete relationships between products and categories.

 

Environment:

  • WebSphere Commerce v 7.0.0.6 FEP5
  • WebSphere Application Server 7.0.0.27
  • DB2 9.7

 

Rate this blog entry:
0
Tagged in: DataLoad Delta Index SOLR Tables delta

Comments

Leave your comment

Guest
Guest Thursday, 28 January 2021

Most Popular Post

WebSphere Commerce, the curious life of a front-end catalog request
Core
Rate this blog entry:
5
36 comment(s) 20185 hits Read More
on 08 October 2014
WebSphere Commerce, the SOLR extension index
Administration
Rate this blog entry:
4
52 comment(s) 17974 hits Read More
on 09 July 2014
WebSphere Commerce, Data Load and SOLR Delta Index
Data Load
Rate this blog entry:
0
128 comment(s) 17677 hits Read More
on 04 June 2013

Tag Cloud

JR47273 Toolkit Dynacache Cores WCUserDistributedMapCache FEP7 CommandLevelAuthorizationCacheCmdImpl Deadlock Admin UI Core WebSphere Commerce Developer InitializationServlet Deploy REST WebSphere Commerce Troubleshooting non-ATP SoapUI preprocess Performances Delta Inventory Quality Validation Data Model Index Web Activity Search Expression Processor Quality Assurance Team Javacore Landing page Thank You Content Store Congratulations SOLR Data cache RAD Coupons Precision Marketing Subcore Search Navigation Flow Management Center ATP Migration WebSphere Application Server WAS Plugin European stores Tables delta Marketing Promotions Search Expression Provider DataLoad ClassNotFoundException

Categories

Latest Blogs

WebSphere Commerce, CommandLevelAuthorizationCache
Cache
Rate this blog entry:
0
Comments (47) Hits (5150)
Posted by Francesco Schettini on 08 December 2016
WebSphere Commerce v8, toolkit exception, ClassNotFound db2
Administration
Rate this blog entry:
1
Comments (36) Hits (6211)
Posted by Francesco Schettini on 18 April 2016
WebSphere Commerce, ATP migration
Store
Rate this blog entry:
0
Comments (37) Hits (7768)
Posted by Francesco Schettini on 28 March 2016
WebSphere Commerce, the curious life of a front-end catalog request
Core
Rate this blog entry:
5
Comments (36) Hits (20185)
Posted by Francesco Schettini on 08 October 2014
WebSphere Commerce, Performance analysis of few European stores
Performance
Rate this blog entry:
0
Comments (27) Hits (5541)
Posted by Francesco Schettini on 22 July 2014