Introduction

This document explains how content flows through the various Pugpig Bolt systems. It should be useful if you are:

• trying to understand the content workflow
• trying to troubleshoot why content might be taking longer than expected to appear on your site or in your app

The stages in this journey are:

• Content is published by editorial
• Content appears in your content feed
• Content is ingested from your content feed into the Express CMS
• The Express CMS notifies the Distribution system that new content is available
• Content is processed by the Distribution system
• Content is visible to the world on the Content Delivery Network
• Content is updated in the Bolt Apps

Appearing in your content feed

It is important you understand any possible delays on your internal systems. Many publishers have various levels of caching, so it might take some time from when editorial publish content to when it is visible in your content feeds. Until it is visible in your feeds, our systems will not be aware of its existence.

Ingesting content into the Express CMS

We will configure the minute polling time between each request. The higher this is, the less load we put onto your server, but the longer it can take for content to update. It is possible to take about to 5 minutes *longer* than this polling time if the system is very busy. If it is taking longer than this, raise a support ticket with us and we can look into it for you.

If this an RSS feed, we can configure this in Express

We also support the RSS syn:frequency tag which lets your feed tell us how often to check.

It is possible that your feed has caches as well, so it might take time for an article to be visible to us in your feed from when you published it.

Note: We do have alternative ways of ensuring updates are quicker - please get in touch if you think you need this. This sometimes involves transmission that does not involve RSS, or your own dedicated systems or queues.

Notification the Express CMS into Pugpig Distribution

Automatic updating (recommended)

Once an article is saved in Express, we normally trigger an update in Distribution automatically and instantly. This is the strongly recommended approach, and is configured in the Pugpig Distribution Service settings plugin in the CMS:

Manual updating

Some clients only want content to be pulled into the Distribution Service manually, so they will use the Update Content and other buttons, available in the 'Distribution Service' tab of the CMS, to do this.

You can also update from the main Distribution Service site which has a few more options, such as updating specific editions.

Scheduled updating (not recommended)

Distribution can also be configured to poll the Express CMS - normally every 5 minutes or so.  This is not recommended, but can be useful if, for example, you want a batch of changes to go live at a specific time of day. This is configured by us using the “cron” syntax (see https://crontab.guru/ for info)

Processing content by Pugpig Distribution

The smaller the size of the change, the faster the content will get into Distribution. You can see the active jobs and the history/timing for every job in the system.

The system is based on queues, so things might take slightly longer if the system is busy. Note that a job that updates many editions at the same time might get placed into a lower priority queue and therefore take longer. This is to ensure that customers who are, for example, loading in back issues do not slow down the customers that are doing normal publishing.

Content available on the Content Delivery Network

Once the content has been pin Distribution, the CDN (Fastly) should be purged automatically and nearly instantly - anything more than 10 seconds would be considered a problem. If new content is visible in Distribution but not the app or site, there may be an issue with the Fastly API we call to purge. If this case, you'll see an error in Distribution that looks something like this: FastlyCDN: <Your App>'s pugpig-ep-prod-<your code> CDN does not exist. One thing to test is a manual Purge via the Purge All button in Distribution.

You can see their status page here: https://status.fastly.com/

Content is available in the Bolt Apps

Once an article or edition is available in Distribution, it should appear in the apps instantly.

However, if an existing article changes, the changes might not be reflected instantly for users that have seen the article before because we cache content on the devices. The length of time we cache it for content for depends on the age of the article. We have three different options here, depending on how important it is for you that updates are quick. Note that the more often we check for updates, the more requests the app will make. This means that performance in the app will be slightly worse, and you'll use slightly more bandwidth.

For end users, it isn’t usually an issue as they very rarely re-read an article they’ve just read in a short amount of time. However, for editorial testing it can be an issue which is why we have a mechanism to clear the app cache (by tapping on the app version number in settings ~10 times, then tapping the 'Clear Downloader Cache' button).

Content retention and expiration

Sometimes, for compliance or rights reasons, you might need to ensure content is no longer available after a certain amount of time. Let us know if this is required and we can discuss the various options.