Updated: June 16, 2021 2:43pm

PrismMQService

(Note: This topic is currently being updated for both Prism 1.14.7 and Prism 2.0; check back later for an update)

PrismMQ is a service that plays an important role in replicating data. PrismMQ is built upon the popular RabbitMQ messaging application. In Prism, the controlling location publishes changes made to records so that they can be picked up by other locations (subscribers). Individual locations subscribe to desired data. The necessary RabbitMQ, Erlang and Prism MQ files are installed when you install the Prism Server.

RMQ Flow

PrismMQService.ini
The PrismMQService.ini file is divided into four sections: Logging, RIL (Retail Pro 9), Prism and General.

When changes to config settings take effect

Config key labels indicate which config settings require a service restart to take effect. In summary, the following protocols apply:

1.    All config settings for Prism services that do not pertain to Log settings, require a service restart to take effect, except as noted in 2.
2.    Config changes for PrismScheduling and PrismTechToolkit services, other than thread counts, timeouts, and reconnect delay,  take effect within 60 seconds of making the change.
3.    Edits to log config settings take effect immediately, without a service restart, with the exception of "Alternate Log Folder". Log folder changes require a restart of all "Retail Pro Prism" services to take effect.

Logging Section
Note: By default, log settings for all Prism services are pulled from the PrismLogging.ini file. If you modify the log settings for PrismMQService in ServiceManager, the changes will override the global settings defined in PrismLogging.ini.

Prism Section Properties

Property Description
Initialization Threads Per Sender
INITTHREADSPERSENDERCNT
How many tasks created when populating Producer_cache table (PrimsMQ) - how many messages are written to the table at a time. The highest number you can set it to is 16.
Initialization Max Senders
INITSENDTHREADCNT
Number of threads that can run simultaneously sending data for initialization. Default = 1 
Initialization Max Receivers
INITRECVTHREADCNT

Thread which picks messages from consumer_cache table and processes them. Setting on initialization is always 1 thread - it doesn't matter what you set it to

Initialization Max Retries
INITMAXRETRIES
NO LONGER USED.
Initialization Batch Size
MESSAGEBATCHSIZE
Default = 25KB    How many messages to read from the database and process at a time. Also affects the logging or progress, as a log entry is only written as each batch is processed (Set to high and memory issues could ensue).
Day-to-Day Threads Per Sender
D2DTHREADSPERSENDERCNT
How many tasks created when populating Producer_cache table (PrimsMQ), or number of tasks for simultaneous process records from producer_cache table to RabbitMQ (PrismMQProducer) ) - how many messages are written to the table at a time. The highest number you can set it to is 16.
Day-to-Day Max Receivers
D2DRECVTHREADCNT
The number of threads which PrismMQConsumer uses for receiving messages from RabbitMQ queue (both initialization and day-to-day) Typically set this (for server mode) to half the number of cores available in the system.
Day-to-Day Max Retries
D2DMAXRETRIES
Default = 5    Maximum number of retries before failing
Day-to-Day Producer Interval
D2DPRODUCERINTERVAL
Default = 10    How often (in seconds) to scan data event queues on Prism side for outgoing changes
Preserve Successful Records
PRESERVESUCCESSRECORDS
True, False    Default = False    Determines whether users will see successful records and errors, or only errors. The default value is False, meaning that successful records are not preserved. Keeping the default value of False is important for customers with large data sets. If set to true, large quantities of unneeded data (sometimes millions of rows) could be preserved. When you run initialization, and everything is successful, but the property is set to false, you will see the completed message but you will not see success records
Initialization Guaranteed Message Delivery
INITGUARANTEEDMESSAGEDELIVERY
If true, messages in RabbitMQ queue are persistent (kept in memory). Should always be true. If set to false, messages will be lost if RabbitMQ crashes. If set to true, RabbitMQ will write messages to files on hard disk.
Resume Initialization at Startup
RESUMEINITONSTARTUP

Receiving side will resume initialization after restart. Has no affect on sender side. Should always be true for receiving side. Only controls the initthreadcount writing messages to the database from the consumer cache table.

Producer Sender Threads
PRODUCERSENDERTHREADS

How many tasks created to send messages to RabbitMQ queue (PrismMQProducer) For initialization - uses 1 no matter how many threads are specified in this setting.

PMQ_PROTECT_ASN

If true and during replication a voucher (which used to be an ASN) is received and an ASN already exists, the service performs consolidation.

Background
In certain scenarios, items added to an ASN can become "lost" during replication. If this setting is enabled, extra protections are applied to ensure that any edits made (e.g. adding items) are maintained correctly when the document is replicated. By default this setting is NOT exposed in the PrismMQService UI. If you experience losing ASN items, add the PMQPPROTECTASN setting (set to True) to the PrismMQService.ini file. If you change the value of the setting, you MUST stop and then restart PrismMQService. If you haven't lost ASN items, do not set the PMQPPROTECTASN setting to True.
Note: If you set the PMQPPROTECTASN setting to True, you will lose ability to REMOVE items from an ASN and replicate the change. If the PMQPROTECTASN setting is set, the receiving side will retain the removed item. For the removed item to be removed at other Prism servers, you must first convert the ASN to a voucher.


General Section
Prism MQ General section

Property Description
Receive Timeout (seconds)
RECEIVETIMEOUTSECONDS
 The number of seconds before a timeout will occur when the machine is waiting to receive messages. Default = 5
Compression Threshold
COMPRESSION
The Compression parameter that helps avoid improve performance when replicating large files, such a large Promotions file. The Compression parameter has a default setting of 100KB. This means that any payload greater than 100KB will be compressed.  Setting the parameter to "0" will turn off compression. A typical 10MB file will compress to about 200KB.
You can leave the default Compression Threshold or modify as needed. Stop and Start the Service after the changes. 0 = Disabled
Server Mode
SERVERMODE
Improves performance for high-performance, multi-CPU systems (four CPU cores or more) when high-throughput and increased processing power are needed.
Enabling Server Mode results in increased CPU usage. We recommend that you only enable Server Mode when it is needed and then disable it when no longer needed.
Server Mode Limitations
Limited to systems with four CPU cores or more.
If you use the same machine as both a server and a workstation, then you probably shouldn't use Server Mode. The Server Mode feature is meant for machines dedicated for use as a server.
By default, the thread counts for Prism and V9 Day-to-Day and Initialization are set to half the CPU Count (CPU Count / 2). This setting ensures that Prism doesn't consume too much of the CPU power when it is first installed.
When Server Mode is enabled, the maximum thread count for Prism and V9 Day to Day and Initialization is set to one less than the total number of CPU cores (CPU Count - 1). Consumer threads will spin up as needed and spin down when no longer needed.
If you later disable Server Mode, the thread counts will be reset to the defaults. (Five threads for Day to Day, 20 for initialization).
Default = False
   
Save Status Interval (D2D Status)
SAVESTATUSINTERVAL

PrismMQ service periodically updates day2day replication status so you can see in the Admin Console what is sent/received. If it is set to 60 (by default), PrismMQ will update status every 60 seconds. is setting controls how often day-to-day replication status will be updated for each connection. PrismMQ updates this status data (number of sent, received and errored messages) in memory and, periodically, starts a background thread that will save those accumulated status updates to the records in the database. Default = 60.

Message Encoding Version
MSGENCODINGVEERSION
As part of the caching feature, Prism 1.14.7 uses a different encoding format for compressed messages.  The MSGENCODINGVERSION (Message Encoding Version) setting in the PrismMQService.ini file controls which encoding format (new or old) is used. This facilitates upgrading to 1.14.7 in a gradual manner. By keeping all servers on the old encoding format, day-to-day replication can continue uninterrupted even if some servers are on 1.14.7 and some are on 1.14.6. To use the old encoding format, set MSGENCODINGVERSION to 0 and restart the PrismMQService. After finishing upgrading all the company's servers, change MSGENCODINGVERSION back to "1." Default = 1.
Host Name
HOSTNAME
Computer machine name (RabbitMQ Host)
Reconnect Delay
RECONNECTDELAY
Number of seconds to wait before attempting to reconnect. Note: Changing this setting requires a restart. Default = 5

Factors that affect PrismMQ Performance

Feature Notes
Anti-virus and Windows defender Add C:\ProgramData\Retailpro folders to the list of exceptions (with sub-folders).
O/S optimization If you change the optimization to "Background services" on the computer that runs the Prism server it will give background services more CPU time and distribute access more evenly. If you are running only Prism Proxy and browser - keep it set to "Programs."
Number of CPU cores and number of background threads

The rule of thumb is  "no more consumer threads than you have CPU cores"; however, this rule applies only to CONSUMER THREADS because they are CPU-intensive. Background threads (like reader threads or sender threads) are different. Background threads (like reader threads or sender threads) can tolerate a higher number of threads than there are CPU cores because those threads use very little CPU processing time. Background threads mostly wait for a response from a database server or from RabbitMQ so running 16 threads on an 8-core server is not a problem; however, eventually you will reach a point of diminishing returns. Therefore, at a maximum set the number of sender threads to 1 or 2 times the number of CPU cores. The total number of sender threads is limited to 16.
Regarding CONSUMER threads, quite often FEWER consumer threads can process messages just as quickly as more. Sometimes TWO consumer threads can even be on par with 8 consumer threads on an 8-core CPU (although it varies from machine to machine). This is because the more CPU-hungry threads are running the more they contend with each other for access to data in memory instead of doing useful work.

Batch Size The batch size affects how the number of reader threads work. Reader threads work on the entire batch, so if your batch size is 20 and the number of reader threads is five, PrismMQ processes five items in parallel (Four groups of five records each). But if your batch size is two while the number of reader threads is five, PrismMQ will be able to utilize only TWO threads (because there are not enough records in the batch to occupy all five threads). On the other hand, setting batch size to some high number will cause problem because it will make PrismMQ to use more memory (the more records are in the batch - the more memory it will take to hold it in memory while those records are getting sent).
Server Mode Increasing the number of background threads will have no effect unless the system has more than two CPU cores and Server Mode is enabled. If the system has fewer than two CPU cores and Server Mode is disabled, little improvement will be gained from increasing the number of background threads.
Number of connections to other Prism systems If there are only a few connections you do not need to change the number of background threads. If there are 10+ connections, you might want to consider using more than one reader and sender threads.
Hard drive write caching policy For drives that hold the C: partition and that host your Prism database. (If you right-click on the DRIVE name in the disk management console and select Properties you will be able to get to the Policies tab. If write caching is enabled it improves hard drive performance which makes RabbitMQ and the database server become more responsive. Warning! This comes with risk. A power failure could result in data loss. We recommend you leave the hard drive write caching policy as is.

Caching of Data during replication

[Note: Caching was introduced in Prism 1.14.7]
Prism caches messages before sending and receiving. This enables replication to proceed more smoothly and helps prevent bottlenecks. Three modules work together to cache replication messages: PrismMQ.exe, PrismMQProducer.exe and PrismMQConsumer.exe.
Sending side
PrismMQService saves messages to the PRODUCER_CACHE table. PrismMQProducer.exe monitors the PRODUCER_CACHE table, sends records it finds to the appropriate subscribers and then deletes the records from the PRODUCER_CACHE table. If the queue has more than 1,000 messages, PrismMQProducer stops sending messages to the queue. PrismMQProducer stops marks the corresponding records in the PRODUCER_CACHE table and adds the names of the paused queues to the REPLICATION_LOCKED_QUEUE table.
Receiving side
PrismMQConsumer.exe monitors the message queue and saves all received messages to  the CONSUMER_CACHE table. PrismMQ.exe monitors the CONSUMER_CACHE table. If PrismMQ.exe finds any messages, it saves those records to the Prism database and deletes them from the CONSUMER_CACHE table. A single queue, Prism.Day2Day, is used for both initialization and day-to-day replication. The INIT column in the PRODUCER_CACHE/CONSUMER_CACHE tables will indicate if the replication was initialization (1) or Day2Day (0).

Restart PrismMQ after Deleting Replication Records
PrismMQ, caches records aggressively to improve performance. If a record is in the cache, it is assumed that the record exists and PrismMQ doesn't need to confirm its existence in the database. This is usually not a problem; however, it can become a problem if you have deleted replication-related records (e.g. replication status) from tables while troubleshooting. The record you removed as part of your troubleshooting efforts may still be in the cache. Therefore, you should restart PrismMQ as soon as possible after deleting replication-related records. Restarting PrismMQ will clear the cache.

Monitoring Total Messages in Queue
PrismMQProducer.exe also monitors the total messages in the queue. If the queue has more than 1,000 messages, it is deemed "clogged" and PrismMQProducer will stop sending messages to the queue. When PrimsMQProducer stops sending messages to a queue, it marks the corresponding records in the PRODUCER_CACHE table and adds the names of the paused queues to the REPLICATION_LOCKED_QUEUE table. Note: There should never be more than 2,000 messages in a queue at any time.
Single Queue for Initialization and Day-to-Day
There is only a single queue now: Prism.Day2Day. This queue is used for both initialization and day-to-day replication. The INIT column in the PRODUCER_CACHE/CONSUMER_CACHE tables will indicate if the replication was initialization (1) or Day2Day (0).

Restart PrismMQ after Deleting Replication Records
The key service involved with replication, PrismMQ, caches records aggressively to improve performance. If the record is in the cache, it is assumed that the record exists and PrismMQ doesn't need to confirm its existence in the database. This is usually not a problem; however, it can become a problem if you have deleted replication-related records (e.g. replication_status) from tables while troubleshooting. The record you removed as part of your troubleshooting efforts may still be in the cache. Therefore, you should restart PrismMQ as soon as possible after deleting replication-related records. Restarting PrismMQ will clear the cache.

RabbitMQ Management Plugin
The rabbitmq-management plugin provides an HTTP-based API for management and monitoring of your RabbitMQ server, along with a browser-based UI and a command line tool, rabbitmqadmin. Features include:

  • Declare, list and delete exchanges, queues, bindings, users, virtual hosts and permissions.
  • Monitor queue length, message rates globally and per channel, data rates per connection, etc.
  • Send and receive messages.
  • Monitor Erlang processes, file descriptors, memory use.
  • Force close connections, purge queues.

 Launch RabbitMQ Management Plugin

  1. In the browser address area, switch from "https" to "http". Clear the port information and anything after. Enter 15672 as the port. Press the Enter key.    
  2. Enter the following login information and then tap or click the Login button: Username: prismguest Password: prismguest    
  3. Select the tab for the area you want to work with.     
  4. When finished, click the Log out button.     

Refer to the following table for a description of the tabs on the RabbitMQ Console:

Tab Description
Overview This tab enables you to see how message processing is progressing over time. The data is displayed in a line chart format. One line chart shows queued messages (Ready, Unacknowledged, Total).
Another line chart shows message rates (Publish, Deliver, Acknowledge, Deliver - no ack).
Connections This tab shows connections. You can filter the list.
Click the HTTP API or Command Line links to display information about using the API.
Channels This tab shows current channels. You can filter the list.
Click the HTTP API or Command Line links to display information about using the API.
Exchanges Select an Exchange in the Name column to view details for the Exchange. The displayed info includes message rate in and message rate out.
Queues View messages and message rates . Messages: Ready, unacknowledged, total; Message rates: (messages per second) Incoming, deliver(get), ack

PrismMQ/RabbitMQ File List

The following files used by PrismMQ are included in the full Prism installation:

File Location Description
RabbitMQ Server …\Program Files (x86)\RabbitMQ Server RabbitMQ is the messaging service upon which PrismMQ is based.
Erlang …\Program Files (x86)\erl6.3 Erlang is the programming language in which RabbitMQ is written. Erlang is required for RabbitMQ.
PrismMQ.exe …\Program Files (x86)\RetailPro\Server\Replication PrismMQ service executable.
PrismMQProducer.exe …\Program Files (x86)\RetailPro\Server\Replication Sending side: PrismMQService saves messages to the PRODUCER_CACHE table. PrismMQProducer.exe monitors the PRODUCER_CACHE table, sends records it finds to the appropriate subscribers and then deletes the records from the PRODUCER_CACHE table. If the queue has more than 1,000 messages, PrismMQProducer stops sending messages to the queue. PrimsMQProducer stops marks the corresponding records in the PRODUCER_CACHE table and adds the names of the paused queues to the REPLICATION_LOCKED_QUEUE table.
PrismMQConsumer.exe …\Program Files (x86)\RetailPro\Server\Replication Receiving side: PrisMQConsumer.exe monitors the message queue and saves all received messages to  the CONSUMER_CACHE table. PrismMQ.exe monitors the CONSUMER_CACHE table. If PrismMQ.exe finds any messages, it saves those records to the Prism database and deletes them from the CONSUMER_CACHE table. A single queue, Prism.Day2Day, is used for both initialization and day-to-day replication. The INIT column in the PRODUCER_CACHE/CONSUMER_CACHE tables will indicate if the replication was initialization (1) or Day2Day (0).
PrismMQService.ini …\ProgramData\RetailPro\Server\conf Contains configuration information for PrismMQ.
PrismMQ.Init.json …\ProgramData\RetailPro\Server\conf Json package file. It is important that you do not modify this file.


ProgramData\RetailPro\Server\RabbitMQ\db

Folder Description
rabbit@[workstation_name]-mnesia Mnesia is a distributed NoSQL database used by RabbitMQ to store information about users, vhosts, exchanges, queues, bindings, index files (the position of messages in queues), and cluster information. RabbitMQ provides its own persistent storage for messages. Persistent messages are stored in the msg_store_persistent directory (both when they are persisted when received on a queue or when memory consumption grows beyond a specific threshold); on the other hand, non-persistent (transient) message are persisted in the msg_store_transient directory (when memory consumption on a queue grows beyond a specific threshold).
rabbit@[workstation_name]-plugins_expand Working directory used to expand enabled plugins when starting the server.

 
ProgramData\RetailPro\Server\RabbitMQ\db rabbit@[workstation_name]-mnesia

Folder Description
msg_store_persistent Messages saved to storage
msg_store_transient Message that have not yet been consumed.
queues    These are the RabbitMQ server components that buffer messages coming from one or more exchanges and send them to the corresponding message receivers.


ProgramData\RetailPro\Server\RabbitMQ\rabbit@[workstation_name]-plugins_expand

Folder Description
amqp_client-3.5.7 This application provides an Erlang library to interact with an AMQP 0-9-1 compliant message broker.
mochiweb-2.7.0-rmq3.5.7-git680dba8 The rabbitmq-mochiweb plugin provides hosting for other plugins that have HTTP interfaces. It allows these interfaces to co-exist on one or more HTTP listeners.
rabbitmq_federation_management-3.5.7 Shows federation status in the management API and UI. Only of use when using rabbitmq_federation in conjunction with rabbitmq_management. In a heterogenous cluster this should be installed on the same nodes as rabbitmq_management.
rabbitmq_federation-3.5.7 Scalable messaging across WANs and administrative domains.
rabbitmq_management_agent-3.5.7 When installing the management plugin on some of the nodes in a cluster, you must install rabbitmq_management_agent on all of the nodes in the cluster. You can install the full management plugin on as many of the nodes as you want.
rabbitmq_management-3.5.7 A management / monitoring API over HTTP, along with a browser-based UI.
rabbitmq_stomp-3.5.7 Provides STOMP protocol support in RabbitMQ.
rabbitmq_web_dispatch-3.5.7 rabbitmq-web-dispatch is a thin veneer around Cowboy that provides the ability for multiple applications to co-exist on Cowboy listeners. Applications can register static docroots or dynamic handlers to be executed, dispatched by URL path prefix.
webmachine-1.10.3-rmq3.5.7-gite9359c7 Associated files