Download code (only if you want to cheat before reading on)

Solr - An old friend

Since this is the holiday season, everyone is talking about their holiday experience. I have just had mine, too, and the highlight of it was the 20-year graduation ceremoney from high school.

37 out of 54 of us got together from all over the world! It was such a joy to see them. We haven't got in touch very often, but when we saw each other again, all the old stories and feelings came back. We have made new friends during the years, it was great, but still nothing can replace the bond with those old ones.

For us who are using Tridion, many have general web development background and/or have used other CMS products before coming to Tridion. Website search is a very common requirement and Solr is a popular choice to drive the search, and it was integrated with all mainstream CMS products. Remind yourself of the days you spent with your old friend Solr - have you tried to tweak its JVM in Production? What did you do to set up a Solr Cloud? What things have you changed in the default solrconfig.xml and schema.xml? Thinking back, aren't they all good memories! Start the video in the beginning of the post, and read on.

Then you start making friends with Tridion, and the joy is, your new friend Tridion has given you a way of keeping in touch with your old friend Solr, through SI4T (Search Integration 4 Tridion, link). It not only can be used to drive free-text searches, but also, with a bit of customisation, a common list and filter requirement can be achieved without having to use more expensive broker queries.

Requirements

The following screenshots show a sample requirement that a global company wants to list their brands, and allows people to filter brand list by category and country.

Solution

• In Tridion, create category "Country" (keyword examples: United Kindom, China, United States, etc.) and "Brand Category" (keyword examples: Food and Drink, Personal Care, Home Care, etc.)
• Create "Brand" schema (fields such as Brand Name, Description, Logo, Country and Brand Category)
• Publish "Brand" components as dynamic component presentations into Solr, with its Country and Category 
• Publish "Country" and "Brand Category" categories into Solr, with each keyword as a solr document
• Solr query to list and filter based on selected "Country" and "Brand Category"

Solr Schemas

The Solr documents for Brand components, each "Country" keyword and each "Brand Category" keyword will look like the following (I removed unncessary ones irrelavent to this post):

Each Brand (Dynamic Component):

{
 "pubdate": "2016-07-26T13:48:15.072Z",
 "id": "dcp:244-413346-410037",
 "brandcategory": "tcm:244-408126-1024",
 "summary": "Brand 1, is a Chinese brand of home care product.",
 "publicationid": 244,
 "typename": "Brand",
 "url": "tcm:244-413346",
 "title": "Brand 1",
 "schemaid": 407661,
 "image": "/Images/brand1_tcm244-408857.gif",
 "type": 407661,
 "country": "tcm:244-407965-1024",
 "itemtype": 16
}

 Each Country (Keyword):

{
 "pubdate": "2015-06-26T13:48:15.072Z",
 "id": "tcm:244-407965-1024",
 "publicationid": 244,
 "typename": "Country",
 "url": "tcm:244-407965-1024",
 "title": "China",
 "type": 50107,
 "itemtype": 1024
}

 Each Brand Category (Keyword): 

{
 "pubdate": "2015-06-22T12:41:35.797Z",
 "id": "tcm:244-408126-1024",
 "publicationid": 244,
 "typename": "Brand Category",
 "url": "tcm:244-408126-1024",
 "title": "Home Care",
 "type": 50110,
 "itemtype": 1024
}

SI4T Customisation

Most of these is done through the OOTB SI4T, except for one. In order to push the keywords into Solr, we will need to create a Deployer Extension to process the published category. This cannot be a Storage Extension as SI4T uses, as there are no APIs in the Storage Extension to get keywords out of a category (you can only publish on category level, not on keyword level)

Code

So to create a Deployer Extension for Taxonomy Deploy:

package com.tridion.deployer.modules
public class SearchExtensionTaxonomyDeploy extends Module {...}

You will need to get the Taxonomy deploy instruction first, then process the items in the instruction (including the category and keywords)

@Override
public void process(TransportPackage data) throws ProcessingException {
  currentPackage = data;
  Section taxonomySection = data.getProcessorInstructions().getSection("Taxonomies");
  if ((setup()) && (taxonomySection != null))
  {
    Iterator iterator = taxonomySection.getFileItems();
    log.debug("Processing section " + taxonomySection.getName());
    while (iterator.hasNext())
    {
      Object item = iterator.next();
      processItem(item, taxonomySection);
    }
  }
}

In the processItem method, get the category root, and look through all keywords in the category and process each keyword (see attached code)

Then for each keyword, you need to construct the XML snippet that SI4T processor understands, which is the same as the OOTB SI4T index data.

private String getKeywordSource(Keyword keyword, TCMURI taxonomyURI, String taxonomyName){
  String keywordURI = keyword.getKeywordURI();
  String keywordName = StringEscapeUtils.escapeXml(keyword.getKeywordName());
  int keywordType = taxonomyURI.getItemId();
  String esTaxonomyName = StringEscapeUtils.escapeXml(taxonomyName);
  //Add publication ID to the search index
  Integer publicationid = taxonomyURI.getPublicationId();

  String keywordIndexDataFormat = "<keyword><!-- INDEX-DATA-START:<indexdata><url>%1$s</url><title>%2$s</title><type>%3$s</type><typename>%4$s</typename><itemtype>1024</itemtype><publicationid>%5$s</publicationid></indexdata>:INDEX-DATA-END --></keyword>";
  return String.format(keywordIndexDataFormat, keywordURI, keywordName, keywordType, esTaxonomyName, publicationid);
}

Then process the keyword using TridionPublishableItemProcessor as normal

protected void processKeyword(Keyword keyword, TCMURI taxonomyURI, String taxonomyName) throws ProcessingException
{
  String keywordSource = getKeywordSource(keyword, taxonomyURI, taxonomyName);
  String uniqueTransactionId = _transactionId;
  TridionPublishableItemProcessor tp = new TridionPublishableItemProcessor(
    keywordSource, FactoryAction.UPDATE, IndexType.TAXONOMY,
    Integer.toString(taxonomyURI.getPublicationId()),
    keyword.getKeywordURI(),
    uniqueTransactionId,
    storageId);
  try {
   tp.processKeywordSource();
  } catch (Exception ex){
   throw new ProcessingException("Could not deploy keyword to search indexer " + keyword, ex);
  }finally{}
}

In TridionPublishableItemProcessor class, add a new processKeywordSource() method by simply calling this.process(). Nothing more.

Similar Deployer Extension for TaxonomyUnDeploy can be created in similar way.

There are things that are less important to explain, but you can download the code and I'm sure you will work out and get a little better acquaintance to your new friend Tridion.

5355.deployer.zip

Configuration 

Besides the Solr configuration in cd_storage_conf.xml, in cd_deployer_conf.xml, register the deployer extension and solr details again

<Deployer>
 <Indexer
  Class="org.si4t.solr.SolrIndexer"
  DefaultCoreUrl="http://localhost:9000/solr/collection-dev"
  Mode="http"
  DocExtensions="pdf,docx,doc,xls,xlsx,pptx,ppt">
  <Urls>
   <!-- SI4T:
     The Value attribute is the complete URL to a Solr Core
     The Id attribute denotes a unique Tridion Publication Id
   -->
   <Url Value="https://dev.solr.com/solr/collection-dev" Id="244" />
  </Urls>
 </Indexer>
  <Processors>
  <Processor Action="Deploy" Class="com.tridion.deployer.Processor">
   <Module Type="SearchExtensionTaxonomyDeploy" Class="com.tridion.deployer.modules.SearchExtensionTaxonomyDeploy" />
            <Module Type="TaxonomyDeploy" Class="com.tridion.deployer.modules.TaxonomyDeploy"/>
  </Processor>
  <Processor Action="UnDeploy" Class="com.tridion.deployer.Processor">
   <Module Type="SearchExtensionTaxonomyUndeploy" Class="com.tridion.deployer.modules.SearchExtensionTaxonomyUndeploy"/>
            <Module Type="TaxonomyUndeploy" Class="com.tridion.deployer.modules.TaxonomyUndeploy"/>
  </Processor>
  </Processors>
</Deployer>

Since you and Solr are old friends, I am assuming you know it well enough, so will not tell you how you can write your Solr query on your site .

So here goes. You can of course get your new friend Tridion to implement all this, but what a joy to pick up something you are so familiar with and you know it will not let you down.

Here is to all my friends and teachers in Class 5,  No. 8 Senior High School, Year '97