Updated: October 21, 2020 9:49am

PrismMQService

PrismMQ is a service that plays an important role in replicating data between RIL and Prism. PrismMQ is built upon the popular RabbitMQ messaging application. In Prism, the controlling location publishes changes made to records so that they can be picked up by other locations (subscribers). Individual locations subscribe to desired data. The necessary RabbitMQ, Erlang and Prism MQ files are installed when you install the Prism Server.

RabbitMQ Management Plugin
The rabbitmq-management plugin provides an HTTP-based API for management and monitoring of your RabbitMQ server, along with a browser-based UI and a command line tool, rabbitmqadmin. Features include:

  • Declare, list and delete exchanges, queues, bindings, users, virtual hosts and permissions.
  • Monitor queue length, message rates globally and per channel, data rates per connection, etc.
  • Send and receive messages.
  • Monitor Erlang processes, file descriptors, memory use.
  • Force close connections, purge queues.

 Launch RabbitMQ Management Plugin

  1. In the browser address area, switch from "https" to "http". Clear the port information and anything after. Enter 15672 as the port. Press the Enter key.    
  2. Enter the following login information and then tap or click the Login button: Username: prismguest Password: prismguest    
  3. Select the tab for the area you want to work with.     
  4. When finished, click the Log out button.     

Refer to the following table for a description of the tabs on the RabbitMQ Console:

Tab Description
Overview This tab enables you to see how message processing is progressing over time. The data is displayed in a line chart format. One line chart shows queued messages (Ready, Unacknowledged, Total).
Another line chart shows message rates (Publish, Deliver, Acknowledge, Deliver - no ack).
Connections This tab shows connections. You can filter the list.
Click the HTTP API or Command Line links to display information about using the API.
Channels This tab shows current channels. You can filter the list.
Click the HTTP API or Command Line links to display information about using the API.
Exchanges Select an Exchange in the Name column to view details for the Exchange. The displayed info includes message rate in and message rate out.
Queues View messages and message rates . Messages: Ready, unacknowledged, total; Message rates: (messages per second) Incoming, deliver(get), ack

Caching of Data during replication
[Note: Caching was introduced in Prism 1.14.7]
Prism caches messages before sending and receiving. This enables replication to proceed more smoothly and helps prevent bottlenecks. Three modules work together to cache replication messages: PrismMQ.exe, PrismMQProducer.exe and PrismMQConsumer.exe.
Sending side
PrismMQService saves messages to the PRODUCER_CACHE table. PrismMQProducer.exe monitors the PRODUCER_CACHE table, sends records it finds to the appropriate subscribers and then deletes the records from the PRODUCER_CACHE table. If the queue has more than 1,000 messages, PrismMQProducer stops sending messages to the queue. PrismMQProducer stops marks the corresponding records in the PRODUCER_CACHE table and adds the names of the paused queues to the REPLICATION_LOCKED_QUEUE table.
Receiving side
PrismMQConsumer.exe monitors the message queue and saves all received messages to  the CONSUMER_CACHE table. PrismMQ.exe monitors the CONSUMER_CACHE table. If PrismMQ.exe finds any messages, it saves those records to the Prism database and deletes them from the CONSUMER_CACHE table. A single queue, Prism.Day2Day, is used for both initialization and day-to-day replication. The INIT column in the PRODUCER_CACHE/CONSUMER_CACHE tables will indicate if the replication was initialization (1) or Day2Day (0).

Restart PrismMQ after Deleting Replication Records
PrismMQ, caches records aggressively to improve performance. If a record is in the cache, it is assumed that the record exists and PrismMQ doesn't need to confirm its existence in the database. This is usually not a problem; however, it can become a problem if you have deleted replication-related records (e.g. replication status) from tables while troubleshooting. The record you removed as part of your troubleshooting efforts may still be in the cache. Therefore, you should restart PrismMQ as soon as possible after deleting replication-related records. Restarting PrismMQ will clear the cache.

Monitoring Total Messages in Queue
PrismMQProducer.exe also monitors the total messages in the queue. If the queue has more than 1,000 messages, it is deemed "clogged" and PrismMQProducer will stop sending messages to the queue. When PrimsMQProducer stops sending messages to a queue, it marks the corresponding records in the PRODUCER_CACHE table and adds the names of the paused queues to the REPLICATION_LOCKED_QUEUE table. Note: There should never be more than 2,000 messages in a queue at any time.
Single Queue for Initialization and Day-to-Day
There is only a single queue now: Prism.Day2Day. This queue is used for both initialization and day-to-day replication. The INIT column in the PRODUCER_CACHE/CONSUMER_CACHE tables will indicate if the replication was initialization (1) or Day2Day (0).

Restart PrismMQ after Deleting Replication Records
The key service involved with replication, PrismMQ, caches records aggressively to improve performance. If the record is in the cache, it is assumed that the record exists and PrismMQ doesn't need to confirm its existence in the database. This is usually not a problem; however, it can become a problem if you have deleted replication-related records (e.g. replication_status) from tables while troubleshooting. The record you removed as part of your troubleshooting efforts may still be in the cache. Therefore, you should restart PrismMQ as soon as possible after deleting replication-related records. Restarting PrismMQ will clear the cache.

PrismMQService.ini
The PrismMQService.ini file is divided into four sections: Logging, RIL (Retail Pro 9), Prism and General.

Logging Section
Note: By default, log settings for all Prism services are pulled from the PrismLogging.ini file. If you modify the log settings for PrismMQService in ServiceManager, the changes will override the global settings defined in PrismLogging.ini.
PrismMQService Logging section
RIL (or Retail Pro 9) section
This section is present when Prism DRS is installed; therefore, it is not present on MySQL systems.

Prism MQ service RIL section

In the following table, the text in uppercase is the text label as it appears in the .ini file.

Property Description
Initialization Batch Size
MESSAGEBATCHSIZE
Positive integer    Default = 25KB    How many messages to read from the database and process at a time. Also affects the logging or progress, as a log entry is only written as each batch is processed (Set to high and memory issues could ensue)
Initialization Max Senders
INITSENDTHREADCNT
 Positive integer    Default = 1    Number of sender threads that can be spun up simultaneously. By default, the setting is set to 1. This setting helps prevent the server from being overwhelmed. If you change the setting, be careful not to set it so high that the server or RabbitMQ are overwhelmed. If you exceed this number, for example by trying to initialize a server while another server's initialization is in progress, the initialization will fail and an error message is written to the log:
Day-to-Day Max Receivers
D2DRECVTHREADCNT
Number of threads that can run simultaneously receiving data for Day to day. Default = 5    
Day-to-Day Max Retries
D2DMAXRETRIES
Maximum number of retries before failing
Default = 5
Preserve Successful Records
PRESERVESUCCESSRECORDS
Determines whether users will see successful records and errors, or only errors (for both initialization and Day to day). The default value is False, meaning that successful records are not preserved. Keeping the default value of False is important for customers with large data sets. If set to true, large quantities of unneeded data (sometimes millions of rows) could be preserved. If set to False, when you run initialization and everything is successful, you will see the completed message in the Status screen but no success records. Default = False
Pub Update Interval
UPDATEPUBTABLESINTERVAL
The number of seconds to wait to scan 1st level DRS logs to catch changes coming from RIL. Default = 100 seconds
Initialization Guaranteed Message Delivery
INITGUARANTEEDMESSAGEDELIVERY

If enabled, PrismMQ takes special steps to ensure that no messages are lost if there is a RabbitMQ failure. If disabled, then some messages may be lost if there is a RabbitMQ Failure (not just a PrismMQ failure). In such a case, even a restart of initialization will likely get stuck and not finish, and the initialization will have to be restarted from the last completed resource.
If you enable Init Guaranteed Message Delivery, then you also should enable Resume Init On Startup.  
By setting both Guaranteed Message Delivery and Resume Init On Startup to "True," if a consumer does down during an initialization, when it is restarted it will resume initialization automatically.
Default = True

Producer Sender Threads
PRODUCERSENDERTHREADS

Controls how many background threads can be used in parallel to put messages in day-to-day replication queues. This is a live setting, so you can change the setting without restarting the service. Simply edit and click Save Settings.
When you set number of sender threads to 1, the multi-threading will not be used in producer thread and the producer thread itself will be sending messages sequentially. That is the default state and it is proper for all stores when they have only one connection to their direct POA.
Default = 1 thread

Prism Section
PrismMQ prism section

Prism Section Properties

Property Description
Initialization Threads Per Sender
INITTHREADSPERSENDERCNT
Default = 5 Set the Initialization Threads Per Sender to 5 or the number of cores you have allocated if you allocate more than 5.
Initialization Max Senders
INITSENDTHREADCNT
 Default = 1    Number of threads that can run simultaneously sending data for initialization.
Initialization Max Receivers
INITRECVTHREADCNT

Default = 20. Number of threads that can run simultaneously receiving data for initialization. Max Threads = 30. If using Server Mode, Max Threads = CPU Count)

Initialization Max Retries
INITMAXRETRIES
 Default = 20    Maximum number of retries - NO LONGER USED.
Initialization Batch Size
MESSAGEBATCHSIZE
Default = 25KB    How many messages to read from the database and process at a time. Also affects the logging or progress, as a log entry is only written as each batch is processed (Set to high and memory issues could ensue).
Day-to-Day Threads Per Sender
D2DTHREADSPERSENDERCNT
 
Day-to-Day Max Receivers
D2DRECVTHREADCNT
Default = 5    Number of threads that can run simultaneously receiving data for Day to day
Day-to-Day Max Retries
D2DMAXRETRIES
Default = 5    Maximum number of retries before failing
Day-to-Day Producer Interval
D2DPRODUCERINTERVAL
Default = 10    How often (in seconds) to scan data event queues on Prism side for outgoing changes
Preserve Successful Records
PRESERVESUCCESSRECORDS
True, False    Default = False    Determines whether users will see successful records and errors, or only errors. The default value is False, meaning that successful records are not preserved. Keeping the default value of False is important for customers with large data sets. If set to true, large quantities of unneeded data (sometimes millions of rows) could be preserved. When you run initialization, and everything is successful, but the property is set to false, you will see the completed message in the V9 Status screen but you will not see success records
Initialization Guaranteed Message Delivery
INITGUARANTEEDMESSAGEDELIVERY
If enabled, PrismMQ takes special steps to ensure that no messages are lost if there is a RabbitMQ failure. If disabled, then some messages may be lost if there is a RabbitMQ Failure (not just a PrismMQ failure). In such a case, even a restart of initialization will likely get stuck and not finish, and the initialization will have to be restarted from the last completed resource.
If you enable Init Guaranteed Message Delivery, then you also should enable Resume Init On Startup. 
By setting both Guaranteed Message Delivery and Resume Init On Startup to "True," if a consumer does down during an initialization, when it is restarted it will resume initialization automatically.
Default = True
Resume Initialization at Startup
RESUMEINITONSTARTUP

If enabled, if a consumer goes down during an initialization and subsequently is restarted, PrismMQ will resume initialization automatically. Be sure to set Initialization Guaranteed Message Delivery to True on the sending server for the desired initialization type (RIL to Prism, or Prism to Prism). If both Resume Init on Startup and Initialization Guaranteed Message Delivery are both set to true, if RabbitMQ goes down, no messages might be lost.
IF Init Guaranteed Message Delivery is False, then some messages may be lost in the event of a RabbitMQ Failure (not just a PMQ failure) and even a restart of initialization will likely get stuck and not finish, and then have to be cancelled and restarted from the last completed resource.
Note: You cannot pause the sender, but each downstream consumer can be paused and/or resumed.
Default = True

Producer Sender Threads
PRODUCERSENDERTHREADS

The number of background threads that can be used in parallel to put messages into day-to-day replication queues. This is a live setting, so you can change the setting without restarting the service. Simply edit and click Save Settings.
When you set number of sender threads to 1, the multi-threading will not be used in producer thread and the producer thread itself will be sending messages sequentially. That is the default state and it is proper for all stores when they have only one connection to their direct POA.
Default = 1

PMQ_PROTECT_ASN

[Note: This setting is not displayed in the user interface; it must be manually added to the .ini file.] In certain scenarios, items added to an ASN can become "lost" during replication. If this setting is enabled, extra protections are applied to ensure that any edits made (e.g. adding items) are maintained correctly when the document is replicated. These changes work for both RIL-to-Prism and Prism-to-Prism replication (both are needed if RIL sends the document to POA and then the POA sends it to the store). By default this setting is NOT exposed in the PrismMQService UI. If you experience losing ASN items, add the PMQPPROTECTASN setting (set to True) to the PrismMQService.ini file. If you change the value of the setting, you MUST stop and then restart PrismMQService. If you haven't lost ASN items, do not add the PMQPPROTECTASN setting.
Note: If you add the PMQPPROTECTASN setting, you will lose ability to REMOVE items from an ASN and replicate the change. If the PMQPROTECTASN setting is set, the receiving side will retain the removed item. For the removed item to be removed at other Prism servers, you must first convert the ASN to a voucher.


General Section
Prism MQ General section

Property Description
Receive Timeout (seconds)
RECEIVETIMEOUTSECONDS
 The number of seconds before a timeout will occur when the machine is waiting to receive messages. Default = 5
Compression Threshold
COMPRESSION
The Compression parameter that helps avoid improve performance when replicating large files, such a large Promotions file. The Compression parameter has a default setting of 100KB. This means that any payload greater than 100KB will be compressed.  Setting the parameter to "0" will turn off compression. A typical 10MB file will compress to about 200KB.
You can leave the default Compression Threshold or modify as needed. Stop and Start the Service after the changes. 0 = Disabled
Server Mode
SERVERMODE
Improves performance for high-performance, multi-CPU systems (four CPU cores or more) when high-throughput and increased processing power are needed.
Enabling Server Mode results in increased CPU usage. We recommend that you only enable Server Mode when it is needed and then disable it when no longer needed.
Server Mode Limitations
Limited to systems with four CPU cores or more.
If you use the same machine as both a server and a workstation, then you probably shouldn't use Server Mode. The Server Mode feature is meant for machines dedicated for use as a server.
By default, the thread counts for Prism and V9 Day-to-Day and Initialization are set to half the CPU Count (CPU Count / 2). This setting ensures that Prism doesn't consume too much of the CPU power when it is first installed.
When Server Mode is enabled, the maximum thread count for Prism and V9 Day to Day and Initialization is set to one less than the total number of CPU cores (CPU Count - 1). Consumer threads will spin up as needed and spin down when no longer needed.
If you later disable Server Mode, the thread counts will be reset to the defaults. (Five threads for Day to Day, 20 for initialization).
Default = False
   
Save Status Interval (D2D Status)
SAVESTATUSINTERVAL

PrismMQ service periodically updates day2day replication status so you can see in the Admin Console what is sent/received. If it is set to 60 (by default), PrismMQ will update status every 60 seconds. is setting controls how often day-to-day replication status will be updated for each connection. PrismMQ updates this status data (number of sent, received and errored messages) in memory and, periodically, starts a background thread that will save those accumulated status updates to the records in the database. Default = 60.

Message Encoding Version
MSGENCODINGVEERSION
As part of the caching feature, Prism 1.14.7 uses a different encoding format for compressed messages.  The MSGENCODINGVERSION (Message Encoding Version) setting in the PrismMQService.ini file controls which encoding format (new or old) is used. This facilitates upgrading to 1.14.7 in a gradual manner. By keeping all servers on the old encoding format, day-to-day replication can continue uninterrupted even if some servers are on 1.14.7 and some are on 1.14.6. To use the old encoding format, set MSGENCODINGVERSION to 0 and restart the PrismMQService. After finishing upgrading all the company's servers, change MSGENCODINGVERSION back to "1." Default = 1.
Host Name
HOSTNAME
Computer machine name (RabbitMQ Host)
Reconnect Delay
RECONNECTDELAY
Number of seconds to wait before attempting to reconnect. Note: Changing this setting requires a restart. Default = 5

Factors that affect PrismMQ Performance

Feature Notes
Anti-virus and Windows defender Add C:\ProgramData\Retailpro folders to the list of exceptions (with sub-folders).
O/S optimization If you change the optimization to "Background services" on the computer that runs the Prism server it will give background services more CPU time and distribute access more evenly. If you are running only Prism Proxy and browser - keep it set to "Programs."
Number of CPU cores and number of background threads

The rule of thumb is  "no more consumer threads than you have CPU cores"; however, this rule applies only to CONSUMER THREADS because they are CPU-intensive. Background threads (like reader threads or sender threads) are different. Background threads (like reader threads or sender threads) can tolerate a higher number of threads than there are CPU cores because those threads use very little CPU processing time. Background threads mostly wait for a response from a database server or from RabbitMQ so running 16 threads on an 8-core server is not a problem; however, eventually you will reach a point of diminishing returns. Therefore, at a maximum set the number of sender threads to 1 or 2 times the number of CPU cores. The total number of sender threads is limited to 16.
Regarding CONSUMER threads, quite often FEWER consumer threads can process messages just as quickly as more. Sometimes TWO consumer threads can even be on par with 8 consumer threads on an 8-core CPU (although it varies from machine to machine). This is because the more CPU-hungry threads are running the more they contend with each other for access to data in memory instead of doing useful work.

Batch Size The batch size affects how the number of reader threads work. Reader threads work on the entire batch, so if your batch size is 20 and the number of reader threads is five, PrismMQ processes five items in parallel (Four groups of five records each). But if your batch size is two while the number of reader threads is five, PrismMQ will be able to utilize only TWO threads (because there are not enough records in the batch to occupy all five threads). On the other hand, setting batch size to some high number will cause problem because it will make PrismMQ to use more memory (the more records are in the batch - the more memory it will take to hold it in memory while those records are getting sent).
Server Mode Increasing the number of background threads will have no effect unless the system has more than two CPU cores and Server Mode is enabled. If the system has fewer than two CPU cores and Server Mode is disabled, little improvement will be gained from increasing the number of background threads.
Number of connections to other Prism systems If there are only a few connections you do not need to change the number of background threads. If there are 10+ connections, you might want to consider using more than one reader and sender threads.
Hard drive write caching policy For drives that hold the C: partition and that host your Prism database. (If you right-click on the DRIVE name in the disk management console and select Properties you will be able to get to the Policies tab. If write caching is enabled it improves hard drive performance which makes RabbitMQ and the database server become more responsive. Warning! This comes with risk. A power failure could result in data loss. We recommend you leave the hard drive write caching policy as is.

PrismMQ/RabbitMQ File List

The following files used by PrismMQ are included in the full Prism installation:

File Location Description
RabbitMQ Server …\Program Files (x86)\RabbitMQ Server RabbitMQ is the messaging service upon which PrismMQ is based.
Erlang …\Program Files (x86)\erl6.3 Erlang is the programming language in which RabbitMQ is written. Erlang is required for RabbitMQ.
PrismMQ.exe …\Program Files (x86)\RetailPro\Server\Replication PrismMQ service executable.
PrismMQProducer.exe …\Program Files (x86)\RetailPro\Server\Replication Sending side: PrismMQService saves messages to the PRODUCER_CACHE table. PrismMQProducer.exe monitors the PRODUCER_CACHE table, sends records it finds to the appropriate subscribers and then deletes the records from the PRODUCER_CACHE table. If the queue has more than 1,000 messages, PrismMQProducer stops sending messages to the queue. PrimsMQProducer stops marks the corresponding records in the PRODUCER_CACHE table and adds the names of the paused queues to the REPLICATION_LOCKED_QUEUE table.
PrismMQConsumer.exe …\Program Files (x86)\RetailPro\Server\Replication Receiving side: PrisMQConsumer.exe monitors the message queue and saves all received messages to  the CONSUMER_CACHE table. PrismMQ.exe monitors the CONSUMER_CACHE table. If PrismMQ.exe finds any messages, it saves those records to the Prism database and deletes them from the CONSUMER_CACHE table. A single queue, Prism.Day2Day, is used for both initialization and day-to-day replication. The INIT column in the PRODUCER_CACHE/CONSUMER_CACHE tables will indicate if the replication was initialization (1) or Day2Day (0).
PrismMQService.ini …\ProgramData\RetailPro\Server\conf Contains configuration information for PrismMQ.
PrismMQ.Init.json …\ProgramData\RetailPro\Server\conf Json package file. It is important that you do not modify this file.


ProgramData\RetailPro\Server\RabbitMQ\db

Folder Description
rabbit@[workstation_name]-mnesia Mnesia is a distributed NoSQL database used by RabbitMQ to store information about users, vhosts, exchanges, queues, bindings, index files (the position of messages in queues), and cluster information. RabbitMQ provides its own persistent storage for messages. Persistent messages are stored in the msg_store_persistent directory (both when they are persisted when received on a queue or when memory consumption grows beyond a specific threshold); on the other hand, non-persistent (transient) message are persisted in the msg_store_transient directory (when memory consumption on a queue grows beyond a specific threshold).
rabbit@[workstation_name]-plugins_expand Working directory used to expand enabled plugins when starting the server.

 
ProgramData\RetailPro\Server\RabbitMQ\db rabbit@[workstation_name]-mnesia

Folder Description
msg_store_persistent Messages saved to storage
msg_store_transient Message that have not yet been consumed.
queues    These are the RabbitMQ server components that buffer messages coming from one or more exchanges and send them to the corresponding message receivers.


ProgramData\RetailPro\Server\RabbitMQ\rabbit@[workstation_name]-plugins_expand

Folder Description
amqp_client-3.5.7 This application provides an Erlang library to interact with an AMQP 0-9-1 compliant message broker.
mochiweb-2.7.0-rmq3.5.7-git680dba8 The rabbitmq-mochiweb plugin provides hosting for other plugins that have HTTP interfaces. It allows these interfaces to co-exist on one or more HTTP listeners.
rabbitmq_federation_management-3.5.7 Shows federation status in the management API and UI. Only of use when using rabbitmq_federation in conjunction with rabbitmq_management. In a heterogenous cluster this should be installed on the same nodes as rabbitmq_management.
rabbitmq_federation-3.5.7 Scalable messaging across WANs and administrative domains.
rabbitmq_management_agent-3.5.7 When installing the management plugin on some of the nodes in a cluster, you must install rabbitmq_management_agent on all of the nodes in the cluster. You can install the full management plugin on as many of the nodes as you want.
rabbitmq_management-3.5.7 A management / monitoring API over HTTP, along with a browser-based UI.
rabbitmq_stomp-3.5.7 Provides STOMP protocol support in RabbitMQ.
rabbitmq_web_dispatch-3.5.7 rabbitmq-web-dispatch is a thin veneer around Cowboy that provides the ability for multiple applications to co-exist on Cowboy listeners. Applications can register static docroots or dynamic handlers to be executed, dispatched by URL path prefix.
webmachine-1.10.3-rmq3.5.7-gite9359c7 Associated files