If your website visitors have to wait a long time for your site to load, at best you will bore them. At worst you will lose their business. This article illustrates how Tridion's extensible storage mechanism enables you to mitigate these risks by integrating with a CDN.

Background

There are many advantages and cost savings in having a centralized CMS and web application server infrastructure, however if your content is viewed by a truely global audience then the perceived performance of your website is out of your control and dependent on the internet connection between your servers and the visitor's browser.

Content Delivery Networks (CDNs) such as Akamai can be used to cache copies of your site content in locations closer to your local markets, distributing both the load on your central infrastructure, providing failover, and more importantly improving the responsiveness of your site thus delivering a better visitor experience.

This article outlines how you can take advantage of the new extensible storage model to integrate with a CDN when publishing and unpublishing content to/from your website. Partial code examples are shown, but it is recommended to download the full example here .

The basics

In the following sections, we will make assumptions to keep things simple, it may well be that these assumptions do not hold in your circumstances. Like any caching mechanism there will always be a trade off between performance, and having a truely up to date site. 

To start we make the following assumptions about the behaviour and functionality of the CDN:

1. The CDN has a central notification mechanism (eg a web service) we can use to tell it that content has been changed/removed
2. The CDN operates by pulling the content directly from your webservers using normal HTTP requests (we are not pushing content directly to their server)

We will also assume (for simplicity) that you are publishing pages and binary content to the filesystem (the same principles apply for the database, but the examples will work with FS storage)

First steps - Where to extend?

The new storage layer gives you the ability to hook into the moment at when a publish transaction (set of content) is committed, so this is exactly what we want to do, when we are sure that the deployment of content is complete, we will notify the CDN what has changed, so it can update it's cache accordingly.

This is done by implementing a custom DAOFactory object, in this case we will simply extend the FSDAOFactory (filesystem storage factory) as we want to still store the content as usual on the filesystem, but additionally notify the CDN, by overriding the commitTransaction method.

public class CDNFSDAOFactory extends FSDAOFactory {

       private  Logger log = LoggerFactory.getLogger(CDNFSDAOFactory.class);

       public void commitTransaction(String transactionId) throws StorageException {

           try

           {

              super.commitTransaction(transactionId);

              notifyCDN(transactionId);

           }

           catch (StorageException storageException) {

               throw storageException;

           }

           catch (Exception cdnException)

           {

              log.error("Error notifying CDN: " + cdnException.getMessage());

           }

           finally {

              cleanupRegister(transactionId);

           }

         }

}

 We will go into the details of the notifyCDN and cleanupRegister methods in the next sections, but this is the basic format for doing the notification.

Handling binaries

The CDNFSDAOFactory class gives us access to the action of committing the transaction, however the actual items being stored are not accessible in this object, for this we need to extend the actual storage classes for each type of object. Lets start with binaries, as the requirements and assumptions required here are simple - when a binary is published, republished or removed we want to notify the CDN accordingly.

We again extend the standard filesystem class FSBinaryContentDAO

public class CDNFSBinaryContentDAO extends FSBinaryContentDAO {

       private  Logger log = LoggerFactory.getLogger(CDNFSBinaryContentDAO.class);

       public void create(BinaryContent binaryContent, String relativePath)  throws StorageException

         {

               super.create(binaryContent, relativePath);

               String transactionId = LocalThreadTransaction.getTransactionId();

               CDNFSDAOFactory.registerAction(transactionId, relativePath, CDNFSDAOFactory.Action.PERSIST);

               log.debug("Created binary so registered PERSIST action: transaction " + transactionId + ", path " + relativePath );

         }

         public void update(BinaryContent binaryContent, String originalRelativePath, String newRelativePath) throws StorageException

         {

               super.update(binaryContent, originalRelativePath, newRelativePath);

                    String transactionId = LocalThreadTransaction.getTransactionId();

                    if (originalRelativePath.compareTo(newRelativePath)!=0)

                    {

                           CDNFSDAOFactory.registerAction(transactionId, originalRelativePath, CDNFSDAOFactory.Action.REMOVE);

                           log.debug("Updated binary moved so registered REMOVE action: transaction " + transactionId + ", path " + originalRelativePath );

                    }

                    CDNFSDAOFactory.registerAction(transactionId, newRelativePath, CDNFSDAOFactory.Action.PERSIST);

                    log.debug("Updated binary so registered PERSIST action: transaction " + transactionId + ", path " + newRelativePath );

         }

         public void remove(int publicationId, int binaryId, String variantId, String relativePath)  throws StorageException

         {

               super.remove(publicationId, relativePath);

                    String transactionId = LocalThreadTransaction.getTransactionId();

                    CDNFSDAOFactory.registerAction(transactionId, relativePath, CDNFSDAOFactory.Action.REMOVE);

                    log.debug("Removed binary so registered REMOVE action: transaction " + transactionId + ", path " + relativePath );

         }

}

Here we are overriding the methods to create, update and remove binaries on the filesystem, in each cased simply calling the super class method, and then registering actions with the CDNFSDAOFactory class static registerAction method (described next). Note that in the case that we do an update (republish) and the binary filename is changed, we need to do both a PERSIST and REMOVE notification to the CDN (tell it to remove the old file, and get the new one)

Using a central register

The CDNFSDAOFactory.registerAction method is used to build up a register of all items within a particular transaction, so that at the point of committing the transaction, we know what items to notify the CDN as being new/updated or removed. For this we use a ConcurrentHashMap:

private static ConcurrentHashMap<String, ConcurrentHashMap<String,Action>> notificationRegister  = new ConcurrentHashMap<String, ConcurrentHashMap<String, Action>>();

       public  static void registerAction(String transactionId, String itemUrl, Action action)

       {

             if (!notificationRegister.containsKey(transactionId))

             {

                    notificationRegister.put(transactionId,new ConcurrentHashMap<String, Action>());

             }

             ConcurrentHashMap<String,Action> transactionActions = notificationRegister.get(transactionId);

             if (!transactionActions.containsKey(itemUrl))

             {

                    transactionActions.put(itemUrl, action);

             }           

             else

             {

                    //Special case where a publish transaction contains a renamed file plus a file

                    //with the same name as the renamed file's old name, we ensure that it is not

                    //removed, but only re-persisted (a rename will trigger a remove and a persist)

                    if (action==Action.PERSIST)

                    {

                           transactionActions.put(itemUrl, action);

                    }

             }

       }

      public static enum Action 
      { 
           PERSIST, REMOVE; 
      }

So for each transaction we build a list of items (using the Url) to notify the CDN as being changed (either a PERSIST (new/updated) or REMOVE (unpublished)). We can then use this in our notifyCDN method, and ensure its kept clean when we are done by implementing a cleanupRegister method:

private  static void cleanupRegister(String transactionId) {

             if (notificationRegister.containsKey(transactionId))

             {

                    notificationRegister.remove(transactionId);

             }

       }

       private void notifyCDN(String transactionId) {

             log.debug("Notifying CDN for transaction" + transactionId);

             if (notificationRegister.containsKey(transactionId))

             {

                    ConcurrentHashMap<String,Action> actions = notificationRegister.get(transactionId);

                    for(String itemUrl : actions.keySet())

                    {

                           log.debug("Notifying CDN for item: " + itemUrl + ", action: " + actions.get(itemUrl));

                    }

             }

       }

There is no CDN notification code here of course, we are simply logging the action - depending on the CDN notification service, this is where you need to plug in some bespoke code.

Configuring it all

With these two extended storage classes, we are ready to deploy our first functionality for notifying for binaries. First we need to create a DAO bundle XML configuration, which describes all the DAO objects which we are going to use. In this case we want to use all the standard FS classes, apart from for binaries

<StorageDAOBundles>

    <StorageDAOBundle type="filesystem">

        <StorageDAO typeMapping="Page" class="com.tridion.storage.filesystem.FSPageDAO"/>

                               <StorageDAO typeMapping="ComponentPresentation" class="com.tridion.storage.filesystem.FSComponentPresentationDAO"/>

                               <StorageDAO typeMapping="Binary" class="com.tridion.extensions.storage.cdn.CDNFSBinaryContentDAO"/>

                               <StorageDAO typeMapping="BinaryVariant" class="com.tridion.storage.filesystem.binaryvariant.FSBinaryVariantDAO"/>

                               <StorageDAO typeMapping="Reference" class="com.tridion.storage.filesystem.FSReferenceEntryDAO"/>

                               <StorageDAO typeMapping="Schema" class="com.tridion.storage.filesystem.FSSchemaDAO"/>

                               <StorageDAO typeMapping="Publication" class="com.tridion.storage.filesystem.FSPublicationDAO"/>

                               <StorageDAO typeMapping="XSLT" class="com.tridion.storage.filesystem.FSXSLTDAO"/>

                               <StorageDAO typeMapping="LinkInfo" class="com.tridion.storage.filesystem.linkinfo.FSLinkInfoDAO"/>

    </StorageDAOBundle>

</StorageDAOBundles>

Save this in your deployer conf folder, and open up the cd_storage_conf.xml file there. In here you can point to your DAO bundle XML,  tell the deployer which DAOFactory to use for filesystem storage, and map the item types to file system storage:

<Configuration Version="6.0">

    <Global>

        <Storages>

            <StorageBindings>

                 <Bundle src="CDNDAOBundle.xml"/>

            </StorageBindings>

            <Storage Type="filesystem" class="com.tridion.extensions.storage.cdn.CDNFSDAOFactory" Id="defaultCdnFile" defaultFilesystem="false">

                <Root Path=" c:\inetpub\live\or\whatever\your\default\filesystem\path\is" />

            </Storage>       

        </Storages>

    </Global>

    <ItemTypes defaultStorageId="filesystem">

         <Item typeMapping="Binary" storageId="defaultCdnFile" cached="true"/>

         //...other mappings in a similar way

    </ItemTypes>

</Configuration>

 Restart your deployer process and you are ready to go.

What about Pages?

We have illustrated the principle of integrating a notification for binaries into the storage layer by extending DAOFactory and BinaryContentDAO classes. You may of course want your CDN to cache pages.

Considerations

If your CDN is to cache pages as well as binaries, we can extend in a similar way, but before that the main consideration is that pages cached will not have truely dynamic functionality (for example dynamic linking, or content delivered by a query) - we need to accept that the cached version will not be representative to benefit from the performance gain. Also some pages simply will not work (for example forms)

This can be mitigated in a number of ways:

1. Dont cache all pages (for example pages with forms, queries or heavy personalization)

2. Ensure the entire cache is periodically refreshed (perhaps every hour for a small site, or every 24 hours for a large site)

3. Give content editors the ability to trigger a cache invalidation of a page or section of the site from the CMS if they notice something isnt right (for example if a dynamically generated news index page is not showing the lastest article, they could trigger a cache invalidation by using a right click GUI extension).

Implementation

The implementation is as simple as the binary extension, this time using the FSPageDAO class:

public class CDNFSPageDAO extends FSPageDAO {

       private  Logger log = LoggerFactory.getLogger(CDNFSPageDAO.class);

       public void create(CharacterData page, String relativePath)    throws StorageException

       {

             super.create(page, relativePath);

             String transactionId = LocalThreadTransaction.getTransactionId();

             CDNFSDAOFactory.registerAction(transactionId, relativePath, CDNFSDAOFactory.Action.PERSIST);

             log.debug("Created page so registered PERSIST action: transaction " + transactionId + ", path " + relativePath );

       }

       public void remove(int publicationId, int pageId, String relativePath)   throws StorageException

       {

             super.remove(publicationId, relativePath);

             String transactionId = LocalThreadTransaction.getTransactionId();

             CDNFSDAOFactory.registerAction(transactionId, relativePath, CDNFSDAOFactory.Action.REMOVE);

             log.debug("Removed page so registered REMOVE action: transaction " + transactionId + ", path " + relativePath );

       }

       public void update(CharacterData page, String originalRelativePath, String newRelativePath) throws StorageException

       {

             super.update(page, originalRelativePath, newRelativePath);

             String transactionId = LocalThreadTransaction.getTransactionId();

             //the super class will do a create, so we don't need to do anything unless the file name has changed

             //(in which case we need to register a remove for the old path)

             if (originalRelativePath.compareTo(newRelativePath)!=0)

             {

                    CDNFSDAOFactory.registerAction(transactionId, originalRelativePath, CDNFSDAOFactory.Action.REMOVE);

                    log.debug("Updated page moved so registered REMOVE action: transaction " + transactionId + ", path " + originalRelativePath );

             }

       }

}

 We then need to configure this class in our DAO bundle, and we are good to go with basic page notifications.

What about Dynamic Component Presentations?

For Dynamic Component Presentations things are more complex. For those which are explicitly put on pages, there is a solution as outlined below. For pages which contain DCP's added by queries, there is no simple solution, and the logic you use to invalidate will depend very much on the types of query you have. We will not address this case in this article.

Outline solution

Key to getting this working is working out where a DCP is used. On the Content Delivery side it is not possible to get this information using the CD API, which means we need to do it when publishing the DCP.

The process will be as follows:

1. When publishing a DCP add a TBB to the Component Template which uses SDL Tridion's where used functionality to find pages which contain this DCP

2. Add a binary (generated on the fly) to the publish transaction which contains this list of pages. We give this a special extension (.dcpinfo)

3. In our extended binary storage class, add a special case to handle the addition and removal of these binaries

Knowing which pages use a DCP

The link at the top of this article contains a download of the source code of this TBB (can be found in GenerateDcpInfo.cs)

Handling the .dcpinfo binary

The following code in your CDNFSBinaryContentDAO will handle the special .dcpinfo files to notify the CDN when DCPs are published/unpublished:

    private  String DCPINFO_EXTENSION = ".dcpinfo";

       public void create(BinaryContent binaryContent, String relativePath)  throws StorageException

         {

               super.create(binaryContent, relativePath);

               String transactionId = LocalThreadTransaction.getTransactionId();

               if (relativePath.endsWith(DCPINFO_EXTENSION))

               {

                      registerDcpInfo(binaryContent,transactionId);

               }

               else

               {

                      CDNFSDAOFactory.registerAction(transactionId, relativePath, CDNFSDAOFactory.Action.PERSIST);

                      log.debug("Created binary so registered PERSIST action: transaction " + transactionId + ", path " + relativePath );

               }

         }

         public void update(BinaryContent binaryContent, String originalRelativePath, String newRelativePath) throws StorageException

         {

               super.update(binaryContent, originalRelativePath, newRelativePath);

               String transactionId = LocalThreadTransaction.getTransactionId();

               if (newRelativePath.endsWith(DCPINFO_EXTENSION))

               {

                      registerDcpInfo(binaryContent,transactionId);

               }

               else

               {

                    if (originalRelativePath.compareTo(newRelativePath)!=0)

                    {

                          CDNFSDAOFactory.registerAction(transactionId, originalRelativePath, CDNFSDAOFactory.Action.REMOVE);

                          log.debug("Updated binary moved so registered REMOVE action: transaction " + transactionId + ", path " + originalRelativePath );

                    }

                    CDNFSDAOFactory.registerAction(transactionId, newRelativePath, CDNFSDAOFactory.Action.PERSIST);

                    log.debug("Updated binary so registered PERSIST action: transaction " + transactionId + ", path " + newRelativePath );

               }

         }

  public void remove(int publicationId, int binaryId, String variantId, String relativePath)  throws StorageException

         {

               super.remove(publicationId, relativePath);

               String transactionId = LocalThreadTransaction.getTransactionId();

               if (relativePath.endsWith(DCPINFO_EXTENSION))

               {

                      File storageRoot = getStorageLocation(publicationId);

                      File target = new File(storageRoot, relativePath);

                      registerDcpInfo(target,transactionId);

               }

               else

               {

                      CDNFSDAOFactory.registerAction(transactionId, relativePath, CDNFSDAOFactory.Action.REMOVE);

                      log.debug("Removed binary so registered REMOVE action: transaction " + transactionId + ", path " + relativePath );

               }

         }

         private void registerDcpInfo(File dcpInfoFile, String transactionId)

         {

               DocumentBuilderFactory docBuilderFactory = DocumentBuilderFactory.newInstance();

          DocumentBuilder docBuilder;

          try {

                    docBuilder = docBuilderFactory.newDocumentBuilder();

                    try {

                           Document doc = docBuilder.parse (dcpInfoFile);

                           registerDcpInfo(doc,transactionId);

                    } catch (Exception e) {

                           e.printStackTrace();

                    }

         }

         private void registerDcpInfo(BinaryContent binaryContent, String transactionId)

         {

               DocumentBuilderFactory docBuilderFactory = DocumentBuilderFactory.newInstance();

          DocumentBuilder docBuilder;

          try {

                    docBuilder = docBuilderFactory.newDocumentBuilder();

                    try {

                           Document doc = docBuilder.parse(new ByteArrayInputStream(binaryContent.getContent()));

                           registerDcpInfo(doc,transactionId);

                    } catch (Exception e) {

                           e.printStackTrace();

                    }

         }

         private void registerDcpInfo(Document doc, String transactionId)

         {

               NodeList listOfPages = doc.getElementsByTagName("page");

          for(int i=0; i<listOfPages.getLength() ; i++)

          {

              Node page = listOfPages.item(i);

              String url = page.getAttributes().getNamedItem("url").getNodeValue();

              CDNFSDAOFactory.registerAction(transactionId, url, CDNFSDAOFactory.Action.PERSIST);

                      log.debug("Found DCP info binary so registered PERSIST action: transaction " + transactionId + ", page " + url );

          }

         }

This is rather lengthy, but essentially we have just updated the create, update and remove methods to check the .dcpinfo file extension, and if so open the file and read the list of page urls to notify.

Note that we always use a PERSIST action here, as when DCPs are unpublished, the page is still there, it just needs to be updated.

Further usage

As you may have noticed, this pattern for extension could have wider usage than just CDN integration. This demostrates a generic method for hooking in notification at the point of committing a publish transaction. Other possible uses for the same pattern include:

• Search engine integration - trigger a search engine to crawl/delete the pages that are updated/removed
• Trigger notifications to reviewers when content is published to staging
• Perform a local cache invalidation when content is published - hooking into whatever caching mechanism you are using in your web application