Fraska Portal

Exploring the WebSphere Commerce world

  • You are here:  
  • Home
  • Blog
  • Francesco Schettini
  • WebSphere Commerce, the "dark side" of the interactions between SOLR delta index and the DataLoad - Part 2
Featured

WebSphere Commerce, the "dark side" of the interactions between SOLR delta index and the DataLoad - Part 2

Posted by on in Data Load
  • Font size: Larger Smaller
  • Hits: 6022
  • 13 Comments
  • Subscribe to this entry
  • Print
WebSphere Commerce, the "dark side" of the interactions between SOLR delta index and the DataLoad - Part 2

 

This article reports another example of interaction between SOLR delta index and data load process.

In particular, If we have to introduce the delta index- in place of the full- and the data load is already set up, or we have to plan from scratch the data load process, we should keep in mind that the data load operations, the order they are executed and the delta updates recorded in the temporary delta tables (TI_DELTA_CATENTRY and TI_DELTA_CATGROUP) are strictly connected; we should not consider the data load a black box respect to the SOLR index, when we need to update the system through the delta mode. The SOLR index and the Data Load are unfortunately tied in a way we cannot think to build a loading process without knowing in which way they interact.

I report two specific cases: 

  1. delete bundles (reported in the previous article);
  2. delete sales categories;

 

Target:

Architects, administrators and developers working on data load and SOLR delta index

Environment:

  • WebSphere Commerce v 7.0.0.6 FEP5
  • WebSphere Application Server 7.0.0.27
  • DB2 9.7

 

Delete sales categories

Delete sales categories using Management Center

The management center allows to delete sales categories- and the underneath relationships with the products- using the "Catalog tool":

 b2ap3_thumbnail_CMC_Delete_SalesCategory_1.png

Once we select the wanted category we can delete it.

b2ap3_thumbnail_CMC_Delete_SalesCategory_2.png

The delete action will delete the category and its relationships with the associated products.

As we already know, this is a basic operations in CMC. However, it's useful understand what happens under the cover when a delete operation is done. As soon as we delete the sales category, the temporary delta tables TI_DELTA_CATENTRY and TI_DELTA_CATGROUP  will be populated in the following way:

TI_DELTA_CATGROUP

  • a row indicating the 'D'- delete operation- of the sales category.

TI_DELTA_CATENTRY

  • a row for each product associated with the deleted category, indicating the 'U'- update operation- of the product;
  • a row for each SKU associated with the deleted category, indicating the 'U'- update operation- of the SKU.

So for example, let's suppose I delete with CMC the sales category 15752 which has just one product associated- 10095- and this product has just one SKU: 10096.

The TI_DELTA_CATENTRY will contain the following data:

MASTERCATALOG_ID;CATENTRY_ID;ACTION;LASTUPDATE
10001;10095;U;2014-04-10 17:28:14.994431
10001;10096;U;2014-04-10 17:28:14.996485

The TI_DELTA_CATGROUP will contain the following data:

MASTERCATALOG_ID;CATGROUP_ID;ACTION;LASTUPDATE
10001;15752;D;2014-04-10 17:28:14.997778

Running the UpdateSearchIndex (in mode = 1, delta mode) the category will properly disappear from the front end, meaning the delta updates is working fine.

So far so good, since we are using the management center. The troubles could come if we use the data load to delete the categories, a really common scenario!

 

Delete sales categories using the Data Load

If we need to remove sales categories we can follow the Info Center indications (keep your attention on "Delete" paragraph):

http://pic.dhe.ibm.com/infocenter/wchelp/v7r0m0/topic/com.ibm.commerce.data.doc/refs/rmlinsertcategory.htm

To make this scenario running we need:

  • an environment XML configuration file;
  • the main XML configuration file;
  • a loader XML configuration file uses two different mediators: CatalogGroupMediator (to load the updates in the catalog) CatalogGroupSearchIndexMediator (to load the updates in the delta tables);
  • a CSV file like the one reported in the Info Center.

Just keep always in mind, if we need to make delta indexes we need to populate also the delta tables, so we need to use also mediators like CatalogGroupSearchIndexMediator during the data load.

You can find the sample files IBM provides in the Data Load sample folder. In particular, we should focus on the loader file; I attach an example (the IBM's one): wc-loader-catalog-group.xml.

You can also find it in your environment under the folder: /xml/config/com.ibm.commerce.catalog/dataload/.

Once again, our aim is to delete the category; running the data load, using- for example- the loader file reported just above, we will see the TI_DELTA_CATENTRY get populated but in a different way than the Management Center does. In fact, this time it will contain just the following data:

TI_DELTA_CATGROUP

MASTERCATALOG_ID;CATENTRY_ID;ACTION;LASTUPDATE
10001;15752;D;2014-04-10 17:28:14.997778

and nothing in the TI_DELTA_CATENTRY.

You see? It's missing the "Update" action of the relationships between products/SKU and the deleted category. 

 

The problem

Despite the data load has followed our indication and deleted the category (in the DB the requested category will be marked as "mark for delete") and the UpdateSearchIndex scheduled job ran in delta mode without any apparent issue, the sales category is still visible in the front end because it's still in the SOLR index indeed, like if nothing happened!

Basically, that happens cause the mediator CatalogGroupSearchIndexMediator does not add the "update" actions of the involved products and SKUs. It had to work exactly like the Management Center but it does not.

 

The solution 

Once we understood they way the search mediators work, we know how we have to set up the dataload. In this specific case, the datalod was made by just a single step:

  1. delete the category

To overcome the mentioned problem, we need to add a further step in the loading process. In particular:

  1. delete the relationships between products and category
  2. delete the category

It means, we will need:

  • an environment XML configuration file;
  • the main XML configuration file;
  • a loader XML configuration file- to map the relationships between category and products/SKUs- uses two different mediators: CatalogGroupAssociationMediator (to load the updates in the catalog) CatalogGroupRelationshipSearchIndexMediator (to load the updates in the delta tables);
  • a loader XML configuration file- to map the categories- uses two different mediators: CatalogGroupMediator (to load the updates in the catalog) CatalogGroupSearchIndexMediator (to load the updates in the delta tables);
  • a CSV file like the one reported in the Info Center.

In this way the data load will process two steps: delete the relationships between products/SKUs and category and than delete the category. The temporary index delta table will be populated exactly as the Management center does, and the SOLR index will be updated properly.

By the way, I found a problem with the CatalogGroupRelationshipSearchIndexMediator; it throws an exception and it does not load the TI_DELTA_CATENTRY as it should. To overcome this problem you could install FEP6 or keep staying on FEP5 installing the iFix JR 47273. 

Rate this blog entry:
0
Tagged in: DataLoad Delta JR47273 SOLR WebSphere Commerce

Comments

Leave your comment

Guest
Guest Thursday, 13 December 2018

Most Popular Post

WebSphere Commerce, the SOLR extension index
Administration
Rate this blog entry:
4
49 comment(s) 14789 hits Read More
on 09 July 2014
WebSphere Commerce, Data Load and SOLR Delta Index
Data Load
Rate this blog entry:
0
121 comment(s) 13557 hits Read More
on 04 June 2013
WebSphere Commerce, the curious life of a front-end catalog request
Core
Rate this blog entry:
5
30 comment(s) 13245 hits Read More
on 08 October 2014

Tag Cloud

Subcore Performances InitializationServlet WebSphere Commerce Developer Tables delta preprocess Landing page ATP Quality Assurance Team Congratulations Management Center Quality Search Expression Provider SOLR ClassNotFoundException Core Web Activity Inventory Troubleshooting Deploy WAS Plugin Store JR47273 Delta Migration European stores Index Coupons Javacore DataLoad WCUserDistributedMapCache Search Expression Processor REST FEP7 Deadlock WebSphere Application Server SoapUI Marketing Admin UI Validation RAD WebSphere Commerce Toolkit Search Navigation Flow Content Data cache CommandLevelAuthorizationCacheCmdImpl Cores Promotions Dynacache Thank You non-ATP Data Model Precision Marketing

Categories

Latest Blogs

WebSphere Commerce, CommandLevelAuthorizationCache
Cache
Rate this blog entry:
0
Comments (41) Hits (2031)
Posted by Francesco Schettini on 08 December 2016
WebSphere Commerce v8, toolkit exception, ClassNotFound db2
Administration
Rate this blog entry:
1
Comments (31) Hits (3536)
Posted by Francesco Schettini on 18 April 2016
WebSphere Commerce, ATP migration
Store
Rate this blog entry:
0
Comments (32) Hits (4827)
Posted by Francesco Schettini on 28 March 2016
WebSphere Commerce, the curious life of a front-end catalog request
Core
Rate this blog entry:
5
Comments (30) Hits (13245)
Posted by Francesco Schettini on 08 October 2014
WebSphere Commerce, Performance analysis of few European stores
Performance
Rate this blog entry:
0
Comments (23) Hits (4178)
Posted by Francesco Schettini on 22 July 2014