retentionPeriod

The "retentionPeriod" property is an optional integer that specifies how many units of data to retain. It refers to the unit of time specified by the "retentionUnit" property — for example, if "retentionPeriod" is 14 and "retentionUnit" is "day", then message data is retained for 14 days. This property defaults to 30.

  • Periodically, the system increments the age of each table partition.
    • "Minute" units are incremented at the zero second of each minute, whether the table partition was created 1 second before or 59 seconds before.
    • "Day" units are incremented at midnight GMT, not midnight of the local UTC time zone. A table partition becomes one day old at midnight GMT, whether it was created one second before or 23 hours and 59 seconds before.
    • "Week" units are incremented at midnight GMT on the last day of each week, even if the table partition was created one second before. The last day of the week is Saturday GMT.
    • "Month" units are incremented at midnight GMT on the last day of each month.
    • "Year" units are incremented at midnight GMT on Dec 31.
  • "retentionPeriod" implicitly calculates an upper bound on how many partitions are able to accumulate on your hard drive. However, partitions are not deleted just because this calculated number of partitions is reached. The system also does not restrict the deletion of all partitions.
  • If the FairCom database is shut down for a month, when the database is started up again, all partitions that are retained for less than one month are immediately deleted.
  • If someone purposely or accidentally sets the date to a future date, all partitions immediately become older, and auto-deletion ensues accordingly.

When partitions are auto-purged, some data are maintained "forever" in a global index. Auto-purge does not prevent these files from growing without bounds and filling up your hard drive.

If not specified, the default found in the services.json file is used. Initially, it is 4 (weeks).

Automatically purging data is important to ensure that retained data does not consume all storage and shut down the computer. The default value of 4 weeks allows FairCom's servers to store 1 TB of messages when 200 topics send one 2K message per second.

Note 

  • If the value is not an integer from 1 to 100, FairCom's servers set it to the default value.
  • Smaller numbers improve SQL performance.
  • Each time the "retentionPeriod" cycles, FairCom's servers automatically and efficiently delete expired data.
  • The "retentionUnit" and "retentionPeriod" properties are used only when the "retentionPolicy" is set to "autoPurge".
  • FairCom's servers only use the "retentionPeriod" property when the "retentionPolicy" is "autoPurge".
  • The "retentionPeriod" can be changed to retain fewer or more messages. Changing it does not necessarily destroy existing data, but data may expire more quickly or be retained longer.
  • The "retentionPeriod" and "retentionUnit" properties control data granularity as well as the retention time. In other words, "retentionPeriod" defines how many sets of data are stored, and "retentionUnit" defines how often data is purged.
    • For example, if "rententionPeriod" is set to 14 , the server stores 14 sets of data. At the beginning of the 15th cycle, the server automatically purges the oldest set of data. If "retentionUnit" is set to day, then data will be purged daily. If set to "week", then data will be purged weekly.
  • The current calendar date affects purging.
    • FairCom's servers automatically purge all retained data that has expired. This is noticeable when FairCom's servers come online after having been offline for a long time. When a server comes back online, it automatically purges all expired data.
    • For example, if a FairCom server is offline for four weeks when it comes online, it will completely purge all retained data that has a retention time of less than 4 weeks.