Store and Forward for high-speed buffering

When configuring the store and forward system for high-speed buffering, you are expecting the case that data will come in quick bursts. By buffering the data, the system can accommodate more information than would be possible going directly against the database.

The key points in configuring a buffering system is to avoid expensive operations like storing and reading from the local cache, and to set the memory buffer large enough to accommodate the expected burst sizes.

Recommended Settings

Memory Buffer

500 or higher. It should be high enough to accommodate several bursts of data. For example, if you expect data to be logged at 100 ms burst for 10 seconds at a time, 100 records would be the minimum value. Data will be forwarded as it comes in, according to the forward settings, but you should not rely on any particular throughput in order to avoid data loss.

Disk Store - Disabled

Depending on your requirements, the disk store should be disabled, or at least set to have high write size/count settings. Writing and reading from the cache is much slower than memory, so it is desirable to avoid it. Of course, the cache should only be disabled if it is ok to lose some data, should the database connection be down for a period of time.

Forward Settings - Write size

Should be larger than the expected burst size. Burst data will be from the same source, and therefore will benefit heavily from the optimizations in the buffer.

Forward Settings - Write Time

Should be balanced in order to give the buffer time to received multiple records that can be optimized, as describe in Write Size above. However, it should not be so long that too much data becomes scheduled to write, which could cause a system slowdown/back up.