How to reduce Lucene Index size in CQ / WEM

Use Case:
1) Index size is huge and it is taking a lot of time for CQ to start.
2) Index size is huge and it is taking a lot of time for Index to rebuild (Also check http://www.wemblog.com/2011/09/how-to-reindex-large-repository.html for this).
3) you use different search service like solr or FAST for indexing
4) you don't use full text searching of documents in site

Solution:

Step 1: Find tika_config file

For version CQ5.4 and less

If you are really not concern about full text indexing of these documents, You could disable indexing of these document in tika-config (crx-quickstart/server/runtime/0/_crx/WEB-INF/classes/org/apache/jackrabbit/core/query/lucene/tika-config.xml). If this folder structure is not present then you have to create one. Original tika_config.xml can be found by unzipping crx-quickstart/server/runtime/0/_crx/WEB-INF/libs/jackrabbit-core-*.jar (Copy it to some other location, rename it to .zip and then unzip) and then going to org/apache/jackrabbit/core/query/lucene).

For version CQ5.5

1) Find the jackrabbit-core jar file and extract the tika config: find ./crx-quickstart/launchpad/felix -name "jackrabbit-core*.jar" | xargs -I {} jar -xvf {} org/apache/jackrabbit/core/query/lucene/tika-config.xml

2) Update org/apache/jackrabbit/core/query/lucene/tika-config.xml file with updated tika file (See step 2)

3) Update the jackrabbit-core jar file with the updated tika-config.xml file: find ./crx-quickstart/launchpad/felix -name "jackrabbit-core*.jar" | xargs -I {} jar -uvf {} org/apache/jackrabbit/core/query/lucene/tika-config.xml

Step 2: Modify file

You could add org.apache.tika.parser.EmptyParser as class for not to parse document type.

For example (To not index excel sheet)

<parser class="org.apache.tika.parser.EmptyParser">
<mime>application/vnd.openxmlformats-officedocument.spreadsheetml.sheet</mime>
</parser>

To remove PDF parsing you can remove entry
<parser name="parse-pdf" class="org.apache.tika.parser.pdf.PDFParser">
<mime>application/pdf</mime>
</parser>

Above method will also help you to reduce Index size (Lucene) in CQ.



Note: To reduce Lucene Index size you can also add following in workspace.xml

<SearchIndex class="com.day.crx.query.lucene.LuceneHandler">
<param name="path" value="${wsp.home}/index"/>
<!-- add below param -->
<param name="supportHighlighting" value="false"/>
</SearchIndex>

As an example please refere this tika-config.xml file.

Some use ful link for tuning your search index in case you can do above,

http://wiki.apache.org/jackrabbit/Search
try to tune "resultFetchSize" and other parameters

Step 3: Disable Indexing using indexing_config.xml file 

Please check instruction below of how to do that. You can add your own node type to reduce index size further. You can use attached indexing_config file

http://dev.day.com/content/kb/home/cq5/CQ5SystemAdministration/SearchIndexingConfig.html

Other Useful Links to reduce lucene Index:

http://dev.day.com/content/kb/home/cq5/CQ5SystemAdministration/how-to-optimize-lucene-index-to-gain-efficiency.html

http://wiki.apache.org/jackrabbit/IndexingConfiguration

http://dev.day.com/content/kb/home/cq5/CQ5SystemAdministration/BoostInSearch.html

http://dev.day.com/content/kb/home/cq5/CQ5Troubleshooting/performancetuningtips.html#TIP05

There is an also ongoing issue to expedite start up time if index size is large https://issues.apache.org/jira/browse/JCR-3107

Important:You have to rebuild index after above changes.



Special thanks to Andrew Khoury and other member from Adobe for sharing information.