(This topic has information about the PrismMQService in Prism 2.x. If using Prism 1.14.7, download the PDF linked above.)
The PrismMQService and its associated programs manage the replication of Prism data between sender and receiver.
This topic has two main sections:
- Background information about the PrismMQService
- Configuration settings for the PrismMQservice.ini file
Consult with a certified Retail Pro technician about your specific data replication needs before making any changes to the PrismMQService.ini file.
About the PrismMQService
The settings in the PrismMQService.ini include settings that enable technicians to:
- Control the outflow of messages from the Producer system during 1) Initialization and separately 2) Day-to-Day replication
- Control the inflow of messages into the Consumer system
- Restart/recover when something goes wrong
- Adjust timers (e.g., SAVESTATUSINTERVAL, HEARTBEAT_INTERVAL)
The PrismMQService is comprised of four modules that work together to allow users to control and fine-tune data replication.
|PrismMQPublisher.exe||Loads records from PUB tables and populates the PRODUCER_CACHE table. This module also sends records during initialization.
Path: Program Files (x86)\RetailPro\Server\Replication
|PrismMQProducer.exe||Monitors the PRODUCER_CACHE table and sends any messages it finds there to the appropriate subscriber, placing the message on the corresponding RabbitMQ queue of that subscriber.
Path: Program Files (x86)\RetailPro\Server\Replication
|PrismMQConsumer.exe||Monitors the corresponding RabbitMQ queue and picks up message that belong to it and places them on the CONSUMER_CACHE table
Path: Program Files (x86)\RetailPro\Server\Replication
|PrismMQ.exe||PrismMQ.exe processes records from the CONSUMER_CACHE table and re-populates PUB tables. PrismMQ.exe also manages the other three executables - PrismMQPublisher.exe, PrismMQProducer.exe and PrismMQConsumer.exe. (Note: When you stop PrismMQ.exe or kill it in Task Manager, PrismMQPublisher, PrimsMQProducer.exe and PrimsMQConsumer.exe are also killed automatically.)
Path: Program Files (x86)\RetailPro\Server\Replication
Sample PrismMQService workflow:
PrismMQService workflow - Sending side
PrismMQPublisher.exe publishes the message and adds it to the PRODUCER_CACHE and PRODUCER_CACHE_DESTINATION tables. The PRODUCER_CACHE table stores the message and the PRODUCER_CACHE_DESTINATION table stores the relational link to each subscriber (destination). PrismMQProducer.exe monitors the PRODUCER_CACHE and PRODUCER_CACHE_DESTINATION tables, and sends records it finds to the appropriate subscribers, placing the message on the appropriate RabbitMQ queue. PrismMQProducer.exe deletes the entry from the PRODUCER_CACHE_DESTINATION table once the message is published on the RabbitMQ queue for that specific destination. Once the message is delivered to each of the subscribers, PrismMQProducer.exe will delete the message contents from the PRODUCER_CACHE table.
Controlling Number of Threads
The PrismMQProducer.exe creates multiple threads for online and offline messages. The number of online threads is controlled by the D2DTHREADSPERSENDERCNT setting (default=5). The number of offline threads is controlled by the OFFLINE_THREADS_CNT setting (default=3).
Controlling Batch Size with PRODUCERBATCHSIZE setting
PrismMQProducer.exe sends 40 records at a time until reaching the defined PRODUCERBATCHSIZE and then moves to the next queue. All the while, the service keeps track of what was sent in memory. Once the defined PRODUCERBATCHSIZE has been sent to all destinations the service will first delete everything marked sent in the destination table and then delete the header (which don't have destination child records). (Note: In Prism 2.1.0 and earlier, the producer sent one message to all queues before moving to the next queue.)
- If the process is interrupted (the user restarts PMQ for example) it will pick up the same set of messages and start over.
- The PRODUCERBATCHSIZE setting can be adjusted some from the default value to improve speed. The higher the number, the longer delete statements will take; the lower the number, delete statements are faster overall it is less efficient and could negatively impact performance. The default PRODUCERBATCHSIZE is 1000 and we recommend keeping it between 1000 and 3000.
- PrismMQProducer.exe monitors the total messages in the queue. If the queue has more than the defined PRODUCERBATCHSIZE number of messages, the queue is deemed "clogged" and PrismMQProducer.exe will stop sending messages to the queue. When PrimsMQProducer.exe stops sending messages to a queue, it marks the corresponding records in the PRODUCER_CACHE table and adds the names of the paused queues to the REPLICATION_LOCKED_QUEUE table.
REPLICATION_LOCKED_QUEUE table STATUS setting
The STATUS setting of the REPLICATION_LOCKED_QUEUE table indicates what is done with a message from the PrismMQProducer.
|NORMAL (status 0)||PrismMQProducer sends the message same as before.|
|LOCKED (status 1)||PrismMQProducer moves the record to the PRODUCER_CACHE_OFFLINE table and removes the payload. PrismMQProducer checks if the record already exists (same queue name, resource name, resource sid and init flag). If the record already exists, PrismMQProducer does not add the record. Instead, the record is removed from the online table. This is how "offline" duplicates are eliminated.
Offline threads work on the PRODUCER_CACHE_OFFLINE table. A separate thread is created for each queue (up to OFFLINE_THREADS_CNT).
If a queue's STATUS is set to "LOCKED," the PrismMQProducer does not create a thread for the queue.
|RESUMING (status 2)||If a queue's STATUS goes back to "UNLOCKED" and there are offline records, the queue is marked as "RESUMING" (status 2) and the offline thread starts to send messages from the Offline table.|
|RESUMED (status 3)||Once the offline thread has sent all offline messages, the queue is marked "RESUMED" (status 3).|
|CLEAR (status 4)||When the online thread sees a queue marked "RESUMED" (status 3) it does one "cycle." It moves messages to the offline table and then marks the queue "CLEARED" (status 4), stops moving records to the offline table and skips one cycle. Skipping a cycle serves as an acknowledgement to sync the offline and online threads for a given queue. The online queue then skips one iteration. This allows the offline thread to send any remaining offline messages. Starting with the next iteration, the queue is marked "NORMAL" (status 0) and the online thread starts sending messages normally.|
PrismMQService Workflow - Receiving side
PrismMQConsumer.exe monitors RabbitMQ and saves all received messages to the CONSUMER_CACHE table. PrismMQ.exe monitors the CONSUMER_CACHE table. If PrismMQ.exe finds any messages, it saves those records to the Prism database and deletes them from the CONSUMER_CACHE table. On the receiving side, a single queue, Prism.Day2Day, is used for both initialization and day-to-day replication. The INIT column in the PRODUCER_CACHE/CONSUMER_CACHE tables will indicate if the replication was initialization (1) or Day2Day (0).
Performance Improvement and Mnesia database size reduction
Disable sending data to systems if they will be offline for more than a few days.
One should avoid killing the erl.exe task or abruptly powering off the system. This can cause corruption of the RabbitMQ Mnesia DB and this can only be resolved by dropping all the queue in the Mnesia DB resulting in a loss of some or all of those messages that were in the queue at that time.
Stop/Start PrismMQService after making changes to config settings
After making changes to the V9, Prism, or General sections of the PrismMQService.ini file, stop the service, wait for it to stop, then start the service again. Do not use the "restart" option.
Configuration information for the PrismMQService are stored in the PrismMQService.ini file.
There are three main sections to the PrismMQService.ini file:
|Prism||The settings in this section apply to Prism-to-Prism replication via initialization or day-2-day.|
|General||General settings including Compression, Reconnect Delay and Save Status Interval|
Prism Section Properties
|Initialization Threads Per Sender
|Positive integer||5||How many tasks created when populating Producer_cache table (PrimsMQ) - how many messages are written to the table at a time.|
|Initialization Max Senders
|Positive integer||1||Defines maximum number of concurrent initialization threads publishing messages on the producer cache.|
|Initialization Max Receiving thread count
Setting is hard-coded to a value of 1 and cannot be adjusted/configured
|Initialization Batch Size
|Positive integer||25||How many messages to read from the database and process at a time. Also affects the logging or progress, as a log entry is only written as each batch is processed (Set to high and memory issues could occur).|
|Day-to-Day Threads Per Sender
|Positive integer||5||Defines the number of threads used by each of the following processes:
|Day-to-Day Max Receivers
This defines the number of threads that PrismMQ.exe and PrismMQConsumer.exe can each use. If you set the value to 5 that means that PrismMQConsumer.exe can use up to 5 threads to move messages from RabbitMQ to the consumer cache and PrismMQ.exe will use up to 5 threads to consume messages. With server mode enabled you can't exceed the number of logical processors and if you do set it to higher Prism will automatically set it to half the number of logical processors.
|Day-to-Day Max Retries
|Positive integer||5||Defines the number of times a message will be reprocessed in the event of an error before it is discarded. If there are issues processing a set of messages a high number here could present a performance issue. However, a low number could result in message being discarded in the event there was some temporary issue such as an Optimistic lock on a record|
|Day-to-Day Producer Interval
|Positive integer||10||Defines how long (in seconds) PrismMQ.exe will wait to scan the PUB_DATA_EVENT table again if it finds the table empty. The frequency increase here can send data more frequently, but it can also result in duplication of messages that results in more overhead.|
|Preserve Successful Records
|True, False||False||Determines whether users will see successful records and errors, or only errors. The default value is False, meaning that successful records are not preserved. Keeping the default value of False is important for customers with large data sets. If set to true, large quantities of unneeded data (sometimes millions of rows) could be preserved. When you run initialization, and everything is successful, but the property is set to false, you will see the completed message but you will not see success records|
|Initialization Guaranteed Message Delivery
|True, False||True||If true, messages in RabbitMQ are flagged as persistent. This will ensure that messages are not deleted in the event RabbitMQ is restarted. False is not recommend, because any reboot or restart of RabbitMQ those messages residing in RabbitMQ will not be retained.|
|Update Publisher tables interval
|Positive integer||105||(1.14.7 and earlier)
Defines frequency (in seconds) that the DRS.DRS* tables are scanned, and the messages are moved to the DRS.PUB* tables. The frequency increase here can send data more frequently but it can also result in duplication of messages that results in more overhead
|Resume Initialization at Startup
If set to TRUE, the Receiving Side (consumer) will resume initialization after the PrismMQ service restarts. Has no affect on sender side. Should always be set to TRUE. This setting only controls the initthreadcount writing messages to the database from the consumer cache table. If set to FALSE, the consumer will not resume initialization upon PrismMQ service restart.
|Producer Sender Threads
[Deprecated in Prism 2.1.x]
(only affects ASNs on receiving side)
|Receive Timeout (seconds)
Defines how long a receive thread will wait before checking to see if PrismMQ is terminating.
The Compression parameter that helps avoid improve performance when replicating large files, such a large Promotions file. The Compression parameter has a default setting of 100. This means that any payload greater than 100KB will be compressed.
|True, False||False||Server Mode enables you to bypass some of the default Windows power management settings so servers can make more efficient use of available computing resources.
Enabling Server Mode results in increased CPU usage. Only enable when needed and then disable when no longer needed.
While enabling server mode on a system can improve the performance of replication it can also have a negative impact. Only set servermode=true if the system has at least 8 or more logical processors. Even on a system with at least 8 or more logical processors, users may need to adjust the number of threads for various settings noted here to see noticeable performance improvement.Use on a temporary basis.
Enabling of server mode is done on a case-by-case basis (depending on the performance of each system). If servermode=true, other integrations or applications on the system may respond poorly or even error out, as this setting prioritizes PrismMQ for CPU processing and other applications can suffer if there aren't sufficient resources available.
Disabling Server Mode will reset thread counts to the defaults. (Five threads for Day to Day, 20 for initialization).
|Save Status Interval (D2D Status)
|Message Encoding Version
|0, 1||1||Used when consuming messages that are compressed. If any system in the enterprise is using old compression method (like 1.14.6 or early version of 1.14.7 the MSGENCODINGVERSION must be set to 0. This facilitates upgrading to 1.14.7 in a gradual manner. By keeping all servers on the old encoding format, day-to-day replication can continue uninterrupted even if some servers are on 1.14.7 and some are on 1.14.6. To use the old encoding format, set MSGENCODINGVERSION to 0 and restart the PrismMQService. After finishing upgrading all the company's servers, change MSGENCODINGVERSION back to 1.|
|REPLICATIONVERSION||Most users should leave this at the default setting. However, there are cases where it is usefult to use this setting. Prism does not guarantee backwards compatibility between versions; thus, RabbitMQ messages may or may not be consumed after upgrade to a newer version of Prism. REPLICATIONVERSION allows users to set a version number on those systems which have been upgraded and leave those on the prior version number until all have been upgraded. This version number is stamped on each message sent and compared at the receiving system with the configuration locally. A system on a lower version number will not process a message stamped with a higher version. Those messages will be stored in the consumer cache until such time the local system was updated and the version number in that configuration file matches or is newer than that stamped on the message.
REPLICATIONVERSION could be used to upgrade a number of system over several days to prevent the loss of messages when the user knows the messages aren't compatible.
1=service gets version from the .ini file. If set to a higher value on sender side and kept at "1" on receiving side, all messages go to receiving-side cache and remain until version on receiving side is equal (or greater).
|string||system||Computer machine name (RabbitMQ Host)|
|Positive integer||5||Defines in RabbitMQ the reconnect delay for all federated queues. Leave this setting at the default value of 5 or less if remote connections are stable and response time is fast.
Typically set to 300 if above is not true
|RECORDSENDDELAY||Positive integer||0||Leave this setting at the default of 0. This was implemented for troubleshooting an issue and has no value at this time to be adjusted.|
|QUEUEMESSAGELIMIT||Positive integer||1000||Sets the message cap allowed per queue in RabbitMQ. When that message cap is reached, PrismMQ will lock the specific queue in the producer cache table.
Default is 1000 and generally speaking this is too low. This setting was hard coded to 1000 in versions prior to 220.127.116.110.
Should be set to 100,000 or greater generally. Locked queues that get large in the producer_cache can create serious performance issues and it is critical to limit locked queue for any length of time. Unlocking queues can take a long time under many circumstances and if the threshold is to low here it could unlock then saturate the queue and then lock the queue again. Additionally because the oldest messages are sent first after unlocking a queue it will send those messages first and if there is a significant number of messages it could delay Day-to-Day messages for the remaining stores impacting every store
Enables Initialization to have priority over day2day. By default, this setting is set to False and thus disabled. If set to True , PrismMQProducer will order outgoing messages by the "Init" flag first. If there are any initialization messages, those messages will go first before any day2day messages. Important! After changing this setting, you must stop the service, wait for it to stop completely, and then start the service again. Do not do a "restart."
|LMDCHECKMODE||True, Falese||True||If TRUE, the Last Modified Date is checked during replication and initialization for the following resources: Customers, Employees, Prism Documents (Sales, Returns and Sales Orders), Vouchers (ASN and Former, Reversed and Reversing Vouchers),
Out Slips (Former, Reversed and Reversing), Adjustment Memos (Former and Reversed), Disbursements, Time Clock, Purchase Orders, Transfer Orders, Vendor, Vendor Invoices
Warning: REPLICATECORERESOURCES should only be used during initial system setup. Setting REPLICATECORERESOURCES to FALSE on an existing production system will jeopardize the integrity of all enterprise data. After the initial join/initialization is complete, it is important to change REPLICATECORERESOURCES back to TRUE (default).
|OFFLINE_THREADS_CNT||Positive integer||3||(2.1.1. and later) The number of threads defined for processing of queues that are offline. Offline queues are triggered by those queues that have exceeded the number of messages allowed in RabbitMQ (defined by QUEUEMESSAGELIMIT). Default = 3|
This is the frequency (number of messages) after which the PrismMQProducer will delete those messages sent to RabbitMQ. The default value is 1000. This means the producer will deliver 1000 messages to each destination queue and once completed, will delete those messages from the producer cache and producer cache destination tables.
|SPECIALDATACHECKINTERVAL||Positive integer||300||(2.x and later) The frequency (in seconds) in which producer checks on the status of special data files being received (e.g. markdowns or PIs). Due to the size of these files, they are replicated differently from typical D2D messages. The file is zipped up and sent in chunks. When the last chunk is received the system will combine them, unzip the file, and then process it. Default = 300|
|QUEUECHECKINTERVAL||Positive integer||60||(2.1.1. and later) The number of seconds the Producer waits between checks to see if it is online of offline. Default = 60.|
Log File Settings in PrismLogging.ini
Note: By default, log settings for all Prism services, including PrismMQService.ini are pulled from the PrismLogging.ini file. If you modify the log settings for PrismMQService, the changes will override the global settings defined in PrismLogging.ini. Edit log settings for PrismMQService in Tech Toolkit Service Manager. When you override the global settings by editing service-specific settings in Service Manager, a [LOG] section is added to the PrismMQService.ini file with only the settings that were overridden. See the Prism Logging topic for more information.
Factors that affect PrismMQService performance
|Anti-virus and Windows defender||Add C:\ProgramData\Retailpro folders to the list of exceptions (with sub-folders).|
|O/S optimization||If you change the optimization to "Background services" on the computer that runs the Prism server it will give background services more CPU time and distribute access more evenly. If you are running only Prism Proxy and browser - keep it set to "Programs."|
|Number of CPU cores and number of background threads||
The rule of thumb is "no more consumer threads than you have CPU cores"; however, this rule applies only to CONSUMER THREADS because they are CPU-intensive. Background threads (like reader threads or sender threads) are different. Background threads (like reader threads or sender threads) can tolerate a higher number of threads than there are CPU cores because those threads use very little CPU processing time. Background threads mostly wait for a response from a database server or from RabbitMQ so running 16 threads on an 8-core server is not a problem; however, eventually you will reach a point of diminishing returns. Therefore, at a maximum set the number of sender threads to 1 or 2 times the number of CPU cores. The total number of sender threads is limited to 16.
|Batch Size||The batch size affects how the number of reader threads work. Reader threads work on the entire batch, so if your batch size is 20 and the number of reader threads is five, PrismMQ processes five items in parallel (Four groups of five records each). But if your batch size is two while the number of reader threads is five, PrismMQ will be able to utilize only TWO threads (because there are not enough records in the batch to occupy all five threads). On the other hand, setting batch size to some high number will cause problem because it will make PrismMQ to use more memory (the more records are in the batch - the more memory it will take to hold it in memory while those records are getting sent).|
|Server Mode||Increasing the number of background threads will have no effect unless the system has more than two CPU cores and Server Mode is enabled. If the system has fewer than two CPU cores and Server Mode is disabled, little improvement will be gained from increasing the number of background threads.|
|Number of connections to other Prism systems||If there are only a few connections you do not need to change the number of background threads. If there are 10+ connections, you might want to consider using more than one reader and sender threads.|
|Hard drive write caching policy||For drives that hold the C: partition and that host your Prism database. (If you right-click on the DRIVE name in the disk management console and select Properties you will be able to get to the Policies tab. If write caching is enabled it improves hard drive performance which makes RabbitMQ and the database server become more responsive. Warning! This comes with risk. A power failure could result in data loss. We recommend you leave the hard drive write caching policy as is.|
RabbitMQ Management Plugin
The rabbitmq-management plugin provides an HTTP-based API for management and monitoring of your RabbitMQ server, along with a browser-based UI and a command line tool, rabbitmqadmin. Features include:
- Declare, list and delete exchanges, queues, bindings, users, virtual hosts and permissions.
- Monitor queue length, message rates globally and per channel, data rates per connection, etc.
- Send and receive messages.
- Monitor Erlang processes, file descriptors, memory use.
- Force close connections, purge queues.
Launch RabbitMQ Management Plugin
- In the browser address area, switch from "https" to "http". Clear the port information and anything after. Enter 15672 as the port. Press the Enter key.
- Enter the following login information and then tap or click the Login button: Username: prismguest Password: prismguest
- Select the tab for the area you want to work with.
- When finished, click the Log out button.
Refer to the following table for a description of the tabs on the RabbitMQ Console:
|Overview||This tab enables you to see how message processing is progressing over time. The data is displayed in a line chart format. One line chart shows queued messages (Ready, Unacknowledged, Total).
Another line chart shows message rates (Publish, Deliver, Acknowledge, Deliver - no ack).
|Connections||This tab shows connections. You can filter the list.
Click the HTTP API or Command Line links to display information about using the API.
|Channels||This tab shows current channels. You can filter the list.
Click the HTTP API or Command Line links to display information about using the API.
|Exchanges||Select an Exchange in the Name column to view details for the Exchange. The displayed info includes message rate in and message rate out.|
|Queues||View messages and message rates . Messages: Ready, unacknowledged, total; Message rates: (messages per second) Incoming, deliver(get), ack|
PrismMQ/RabbitMQ File List
The following files used by PrismMQ are included in the full Prism installation:
|RabbitMQ Server||…\Program Files (x86)\RabbitMQ Server||RabbitMQ is the messaging service upon which PrismMQ is based.|
|Erlang||…\Program Files (x86)\erl6.3||Erlang is the programming language in which RabbitMQ is written. Erlang is required for RabbitMQ.|
|PrismMQ.exe||…\Program Files (x86)\RetailPro\Server\Replication||PrismMQ service executable.|
|PrismMQProducer.exe||…\Program Files (x86)\RetailPro\Server\Replication||Sending side: PrismMQService saves messages to the PRODUCER_CACHE table. PrismMQProducer.exe monitors the PRODUCER_CACHE table, sends records it finds to the appropriate subscribers and then deletes the records from the PRODUCER_CACHE table. If the queue has more than 1,000 messages, PrismMQProducer stops sending messages to the queue. PrimsMQProducer stops marks the corresponding records in the PRODUCER_CACHE table and adds the names of the paused queues to the REPLICATION_LOCKED_QUEUE table.|
|PrismMQConsumer.exe||…\Program Files (x86)\RetailPro\Server\Replication||Receiving side: PrisMQConsumer.exe monitors the message queue and saves all received messages to the CONSUMER_CACHE table. PrismMQ.exe monitors the CONSUMER_CACHE table. If PrismMQ.exe finds any messages, it saves those records to the Prism database and deletes them from the CONSUMER_CACHE table. A single queue, Prism.Day2Day, is used for both initialization and day-to-day replication. The INIT column in the PRODUCER_CACHE/CONSUMER_CACHE tables will indicate if the replication was initialization (1) or Day2Day (0).|
|PrismMQService.ini||…\ProgramData\RetailPro\Server\conf||Contains configuration information for PrismMQ.|
|PrismMQ.Init.json||…\ProgramData\RetailPro\Server\conf||Json package file. It is important that you do not modify this file.|
|rabbit@[workstation_name]-mnesia||Mnesia is a distributed NoSQL database used by RabbitMQ to store information about users, vhosts, exchanges, queues, bindings, index files (the position of messages in queues), and cluster information. RabbitMQ provides its own persistent storage for messages. Persistent messages are stored in the msg_store_persistent directory (both when they are persisted when received on a queue or when memory consumption grows beyond a specific threshold); on the other hand, non-persistent (transient) message are persisted in the msg_store_transient directory (when memory consumption on a queue grows beyond a specific threshold).|
|rabbit@[workstation_name]-plugins_expand||Working directory used to expand enabled plugins when starting the server.|
|msg_store_persistent||Messages saved to storage|
|msg_store_transient||Message that have not yet been consumed.|
|queues||These are the RabbitMQ server components that buffer messages coming from one or more exchanges and send them to the corresponding message receivers.|