How to improve Tar Optimization and Indexing process in CQ 5.4 / WEM

Use Case: You have a lot of tar files (Because of some bad code) that you want to clear quickly Or you want to improve indexing process while reindexing repository.

Pre-requisite CRX2.2 with Hotfix 2.2.0.46 or greater.

Solution

There are two new TarPersistenceManager configuration
options.

The optimizeCount option controls how many blocks the
TarPM optimizer processes at a time. The default value is
one, which avoids blocking concurrent user operations as
much as possible. By increasing this value the TarPM
optimization speed increases, but the performance of
concurrent user operations decreases.

In addition to this you can do following,

The main problem of the Tar PM optimization is reading from the Tar index files in random order. To speed this up, load the files in the buffer cache every few minutes, from both the crx.default and the version directory:
cat .../repository/workspaces/crx.default/index*.tar > /dev/null
cat .../repository/version/index*.tar > /dev/null


The indexInMemory option is a boolean flag for making the
repository read all TarPM index files entirely in memory.
This can significantly boost performance for large
repositories, but requires gigabytes of free RAM.

Here is How this needs to be set,

<PersistenceManager class="com.day.crx.persistence.tar.TarPersistenceManager">
<param name="optimizeCount" value="{Numeric value you want to set}" />
<param name="indexInMemory" value="true" />
</TarPersistenceManager>


Please test it before use

Special Thanks to Thomas Mueller and Jukka zitting from Adobe for this information.