How to check if some one has replication rights for a path in CQ5 / WEM

You can use following code

import javax.jcr.security.AccessControlManager;
import javax.jcr.security.Privilege;
import com.day.cq.replication.Replicator;
import javax.jcr.Session;

public static boolean canReplicate(String path, Session session) throws RepositoryException {
AccessControlManager acMgr = session.getAccessControlManager();
return session.getAccessControlManager().hasPrivileges(path, new Privilege[] {acMgr.privilegeFromName(Replicator.REPLICATE_PRIVILEGE)});
}

In fact you can use this code to check any rights. Just replace privilege with any privilege in http://jackrabbit.apache.org/api/1.5/org/apache/jackrabbit/api/jsr283/security/Privilege.html

For example


privilegios[0] = acMgr.privilegeFromName(Privilege.JCR_READ);
privilegios[1] = acMgr.privilegeFromName(Privilege.JCR_ADD_CHILD_NODES);
privilegios[2] = acMgr.privilegeFromName(Privilege.JCR_REMOVE_CHILD_NODES);
.. More

How to rebuild index in CQ5 / WEM

Prerequisite: You have to stop your CQ instance (Except for AEM6 onward)

Also read

http://www.wemblog.com/2011/09/which-index-is-what-in-cqcrx.html

http://www.wemblog.com/2011/09/how-to-reindex-large-repository.html

For CQ5.2X or CQ5.3 or upgraded CQ5.4 remove or rename following files or folder

1) /crx-quickstart/repository/version/copy/index*.tar
2) /crx-quickstart/repository/workspaces/crx.default/copy/index*.tar
3) /crx-quickstart/repository/workspaces/crx.default/index
4) /crx-quickstart/repository/repository/index


In some cases you can also recreate indexes by deleting or renaming following folders and files (Some of the files and folder has nothing to do with index, they are just removed for cleaning purpose

1) /crx-quickstart/repository/shared/journal
2) /crx-quickstart/repository/workspace/crx.default/index
3) /crx-quickstart/repository/workspace/crx.default/copy
4) /crx-quickstart/repository/workspace/version/copy/index*
5) /crx-quickstart/repository/index
6) /crx-quickstart/repository/shared/journal/control
7) /crx-quickstart/repository/shared/version/control
8) /crx-quickstart/repository/shared/workspace/crx.default/control
9) /crx-quickstart/logs

For CQ5.4 / CRX2.2 / CQ5.5 / CRX2.3 remove or rename following files and folders

1) /crx-quickstart/repository/workspaces/crx.default/index
2) /crx-quickstart/repository/workspaces/crx.default/index*.tar
3) /crx-quickstart/repository/version/index*.tar
4) /crx-quickstart/repository/tarJournal/index*.tar
5) /crx-quickstart/repository/repository/index

For AEM6 onward (If you are using oak MK), Rebuilding indexes can happen while system is running (Note that query would not run faster if you do reindex)

Please read https://docs.adobe.com/docs/en/aem/6-0/deploy/upgrade/queries-and-indexing.html and https://docs.adobe.com/docs/en/aem/6-1/deploy/best-practices/best-practices-for-queries-and-indexing.html before doing re indexing.

I think best way to do reindexing in AEM6 is to use this tool http://adobe-consulting-services.github.io/acs-aem-commons/features/oak-index-manager.html

for doing full indexing of repo in AEM6 you can set reindex to true at oak:index node. More info is here http://jackrabbit.apache.org/oak/docs/query.html

If you don't have access to install this tool then you can manually go to crxde repo and set reindex property to true for property you want to reindex (This is expensive operation, don't do it if you don't know what you are doing).

In some cases you need to stop your system remove/rename repository/index folder and then start instance again

Note: In some cases if your repository size is huge, It could take days to rebuild Lucene index. So Before you rebuild your index, make sure that you can afford that much of downtime.

How to add a supported language to the CQ / WEM

1) You need to overlay/overwrite /libs/cq/security/widgets/source/widgets/security/Preferences.js to change the list of available languages (used in security admin)
and overlay /libs/cq/security/content/tools/userProperties (inside change language list under items/common/items/lang/options - similar dialog to Preferences.js) (used for user drop-down in the top right on admin consoles)

2) Then you need to add the new language somewhere in the repo, e.g. /apps/yourapp/i18n, following the layout as under /libs/wcm/core/i18n (see also the above section on the translator UI). Stuff under /libs must not be changed!

3) Here is a sample package:
cq-i18n-add-language-sample.zip

4) adds /apps/test/i18n with translation of "New..." in zh-cn ("simplified") and zh-tw ("traditional") languages
overlays /apps/cq/security/content/tools/userProperties and /apps/cq/security/widgets/source/widgets/security/Preferences.js to add the zh-tw language

5) to check, set the current user's language to "Traditional Chinese" and look in the siteadmin: the first button in the toolbar should now be "traditional" instead of "New..."

Thanks Alexander Klimetschek from Adobe for this information.

How to use multi language translation in JSP in CQ / WEM

Use Case: You want to display certain text in page based on page language

Prerequisite: http://sling.apache.org/site/internationalization-support-i18n.html

Solution: You can use i18n bundle approach for this

Suppose you want to display certain text for a component in different language based on page language or URL. Take an example of /apps/geometrixx/components/homepage component.

1) Create a i18n node under /apps/geometrixx/components/homepage of type sling:Folder
2) create a node of type sling:Folder under i18n of mixin type mix:language
3) Add a property called jcr:language and assign value of lang code or lang_COUNTRY

Structure would be like

/apps/myApp
+-- English (nt:folder, mix:language)
| +-- jcr:language = en
| +-- mx (sling:messageEntry)
| +-- sling:key = "msgXXX"
| +-- slign:message = "An Application Text"
+-- Deutsch (nt:folder, mix:language)
+-- jcr:language = de
+-- mx (sling:messageEntry)
+-- sling:key = "msgXXX"
+-- slign:message = "Ein Anwendungstext"



4) Then on jsp page use following code

<cq:setContentBundle/>
<%
Locale pageLocale = currentPage.getLanguage(false);
//If above bool is set to true. CQ looks in to page path rather than jcr:language property.
ResourceBundle resourceBundle = slingRequest.getResourceBundle(pageLocale);
I18n i18n = new I18n(resourceBundle);
%>
I am printing using I18n <%=i18n.get("updateLocations") %>
I am printing using fmt:message now <fmt:message key="updateLocations"/>
I found page locale is <%=pageLocale.toString() %>

You can also test language using grid under http://localhost:4502/libs/cq/i18n/translator.html
More Info: http://labs.sixdimensions.com/blog/dklco/2012-05-25/adobe-cq5-translator

To change what languages are shown in the translator's grid, create a multi-value string property /etc/languages/languages (node structure must be created), containing the iso language codes (e.g. ["de", "fr", …]). The default is hard coded in the translator and maps the ootb CQ languages: ['de', 'fr', 'it', 'es', 'ja', 'zh-cn']


Some More Good blog Post on this:

http://blogs.adobe.com/dekesmith/2012/10/21/internationalization-within-sling-and-cq

How to manage bundle using curl command

Get bundle Symbolic Name:

(You can use jsawk tool to parse this data as well https://github.com/micha/jsawk)

curl -u admin:admin http://localhost:4502/system/console/bundles</bundlenumber>.json

Start
curl -u admin:admin -F action=start http://localhost:4502/system/console/bundles/$bundle symbolic name

Stop
curl -u admin:admin -F action=stop http://localhost:4502/system/console/bundles/$bundle symbolic name

Update
curl -u admin:admin -F action=update http://localhost:4502/system/console/bundles/$bundle symbolic name

Refresh
curl -u admin:admin -F action=refresh http://localhost:4502/system/console/bundles/$bundle symbolic name

Unistall
curl -u admin:admin -F action=uninstall http://localhost:4502/system/console/bundles/$bundle symbolic name
Or
curl -XDELETE -u user:pass http://.../apps/${app}/install/${bundle-file-name}

Install
curl -X POST -v -F action=install -F bundlestart=start -F bundlestartlevel=20 -F bundlefile=/path/to/jar/file.jar http://admin:admin@localhost:4502/system/console/bundles
OR
curl -u admin:admin -F action=install -F bundlestartlevel=20 -F bundlefile=@"/path/of/jar" http://localhost:9002/system/console/bundles


Here is some example with jswak (Here is how you can install jswak on MAC)

• http://www.macports.org/install.php
• http://johnattebury.com/blog/2011/06/spidermonkey-jsawk-resty-on-snow-leopard

How to check if a CQ bundle is Active

curl -u admin:admin http://localhost:4502/system/console/bundles/<bundle number or symbolic name>.json | jsawk -n 'if(this.data[0].state=="Active")' | echo "Bundle is Active"


How to check if All bundles are active

curl -u admin:admin http://localhost:4502/system/console/bundles/<bundle number or symbolic name>.json | jsawk -n 'if(this.s[3] + this.s[4] > 0)' | echo "There is some problem not all bundle is active"

Find all Bundle having some problems

curl -u admin:admin http://localhost:4502/system/console/bundles.json | jsawk -n 'for(var i=0;i<this.data.length;i++) {if(this.data[i].state!="Active" && this.data[i].state!="Fragment") out("Bundle " + this.data[i].symbolicName+"  is "+ this.data[i].state)}'


 

How to transfer file from one CRX to other modified after certain date using CURL in CQ / WEM / CRX

Use case: You want to copy all files modified after certain date and time from one instance to another. this is also use ful for migration projects.

Solution You can use following unix script to do that (Example refer /var/dam) change it based on your requirement

Approach: Initially I thought, I will build list of all the pages and then use vlt rcp. But problem with this approach is, on destination you might not have Path (Subfolder) and in that case you will get path not found exception using vlt. Instead you can use reciprocal approach and find all pages created "before" certain date and then exclude them (Good for smaller size folder, not good for huge data) from all vlt copy from parent folder (Make sure that that exist in destination).

Prerequisite make sure that you have VLT configured. Please see http://dev.day.com/docs/en/crx/current/how_to/how_to_use_the_vlttool.html for that

Solution

#!/bin/bash
# Author: upadhyay.yogesh@gmail.com

# The host and port of the source server
SOURCE="localhost:4504"

# The user credentials on the source server (username:password)
SOURCE_CRED="admin:admin"

# The host and port of the target server
TARGET="localhost:4508"

# The user credentials on the target server (username:password)
TARGET_CRED="admin:admin"

#Filter path
FILTER_PATH="/var/dam"

#Root Path
ROOT_PATH="/var/dam"

FILTER_COUNT=0

SPACE=" "
#############################################################################

#Query to get all files uploaded after certain date
#echo "Creating path list"
ALL_PATHS=`curl -s -u $SOURCE_CRED "$SOURCE/bin/querybuilder.json?path=$FILTER_PATH&type=nt:file&&p.limit=-1&daterange.property=jcr:created&daterange.upperBound=2011-11-11&daterange.upperOperation=<&orderby=jcr:created&orderby.sort=desc" | tr ",[" "\n" | grep path | awk -F \" '{print $4 "\n"}'`
echo "$ALL_PATHS"
for SINGLE_PATH in $ALL_PATHS
do
EXCLUDE=${EXCLUDE}${SINGLE_PATH}${SPACE}
done
vlt rcp -e '$EXCLUDE' -r http://$SOURCE_CRED@$SOURCE/crx/-/jcr:root$ROOT_PATH http://$TARGET_CRED@$TARGET/crx/-/jcr:root$ROOT_PATH
echo "vlt rcp -e \"$EXCLUDE\" -r http://$SOURCE_CRED@$SOURCE/crx/-/jcr:root$ROOT_PATH http://$TARGET_CRED@$TARGET/crx/-/jcr:root$ROOT_PATH"

How to use CURL command to find pending and blocking job in the replication Queue in CQ5.4 / WEM

Use case : You want to write monitoring script to find all pending and blocking job in the replication queue and do some action.

Solution : Use following curl commands (Change server name, Port, UID and PWD)

To find all pending Jobs
curl -s -u admin:admin "http://localhost:4504/bin/querybuilder.json?&path=/var/eventing/jobs&type=slingevent:Job&p.limit=-1&fulltext=/com/day/cq/replication/job&fulltext.relPath=@slingevent:topic&property.and=true&property=slingevent:finished&property.operation=not&orderby=slingevent:created&orderby.sort=asc" | tr ",[" "\n" | grep path | awk -F \" '{print $4 "\n"}'

To find all Blocking Jobs
curl -s -u admin:admin "http://localhost:4504/bin/querybuilder.json?path=/var/eventing/jobs/anon&type=slingevent:Job&rangeproperty.property=event.job.retrycount&rangeproperty.lowerBound=1" | tr ",[" "\n" | grep path | awk -F \" '{print $4 "\n"}'

Once you have blocking jobs, You can go to CRX and remove blocking entry (Before that make sure that blocking entry is causing problem).

You can also use replication clean up script (It is custom script I wrote to remove one entry) to remove one entry from the queue and then if necessary activate them again.

Blocking replication queue can happen because,

1) There is some problem with sling eventing and queue is not getting processed (For that restart sling event support bundle from felix console)

2) There is a blocking job in the queue (For that find blocking entry in the queue using above curl command and remove it)

3) There is some problem with publish server (503 or OOM etc, In that case restarting publish server should resolve the issue)

How to change all password using curl in CQ / WEM (Till CQ5.4)

=====================================
#!/bin/bash

OLD_PWD=admin
NEW_PWD=admin
CONSOLE_PWD=admin
HOST=localhost
PORT=4502
echo changing crx admin password:
curl --data rep:password=$NEW_PWD --user admin:admin http://$HOST:$PORT/home/users/a/admin

echo change cqse admin password:
curl --data username="admin" --data password_old=$OLD_PWD --data password=$NEW_PWD --data password_check=$NEW_PWD --user admin:admin http://$HOST:$PORT/admin/passwd

echo change replication agent password:
curl --data transportPassword=$NEW_PWD --user admin:$OLD_PWD http://$HOST:$PORT/etc/replication/agents.author/publish/jcr:content

echo change sling crx client password:
echo below script does:
echo 1. find specific pid (needs https://github.com/micha/jsawk)
echo 2. set password property for that pid
FACTORY_PID=com.day.crx.sling.client.impl.CRXSlingClientRepository

PID=`curl -s -u admin:$CONSOLE_PWD "http://$HOST:$PORT/system/console/configMgr/(service.factoryPid=$FACTORY_PID).json" | jsawk -n 'out(this.pid)'`
echo ">> $PID"
curl -s -u admin:$CONSOLE_PWD -dapply=true -dadmin.password=$NEW_PWD -dpropertylist=admin.password http://$HOST/system/console/configMgr/$PID > /dev/null

=====================================


Note: Have not tested it. Please test it before use.

Thanks Alexander Klimetschek from Adobe for this information.

How to truncate and interpret Tar File in CQ / WEM

Use Case: You want to analyze what is there in tar file or possibly truncate it using tar file truncate tools.

Solution:

Soppose you try to analyzing very first data tar file under (/crx-quickstart/repository/workspace/crx.default)

$ tar -tvf data_00010.tar | less

-rwxrwxrwx 0/0 301 2011-11-22 02:00 cafebabe-cafe-babe-cafe-babecafebabe.na
-rwxrwxrwx 0/0 80 2011-11-22 02:00 deadbeef-cafe-babe-cafe-babecafebabe.na
-rwxrwxrwx 0/0 142 2011-11-22 02:00 f34600a5-be47-41be-8568-8c79dcbd7355.na
-rwxrwxrwx 0/0 296 2011-11-22 02:00 21232f29-7a57-35a7-8389-4a0e4a801fc3.na
-rwxrwxrwx 0/0 348 2011-11-22 02:00 a9c3313a-c296-486d-99ac-b56ebfdefd26.na
-rwxrwxrwx 0/0 192 2011-11-22 02:00 63781579-2ec1-441d-b582-3eea7ce6a0bd.na
-rwxrwxrwx 0/0 276 2011-11-22 02:00 294de355-7d9d-30b3-92d8-a1e6aab028cf.na
-rwxrwxrwx 0/0 286 2011-11-22 02:00 984c30b3-96f5-48b7-a1f3-3d74d0f00460.na
-rwxrwxrwx 0/0 0 2011-11-23 11:30 1322065821400/2782dd9a-c5a8-4689-a9aa-ae46a4074ac3.n
.....
-rwxrwxrwx 0/0 20 2011-11-22 02:00 commit-1321945201521.sh

Entry without <number>/<UUID> is system UUID (First one being root node) and should not be modified.

The entries with 1322065821400/2782dd9a-c5a8-4689-a9aa-ae46a4074ac3.n mean the transaction was started at Wed Nov 23 2011 11:30:21 GMT-0500 (EST) (= System.out.println(new java.sql.Timestamp(1299071667963L).toString()) or http://www.esqsoft.com/javascript_examples/date-to-epoch.htm) on UUID 2782dd9a-c5a8-4689-a9aa-ae46a4074ac3 then

entry with commit-1321945201521.sh mean transaction committed at Tue Nov 22 2011 02:00:01 GMT-0500 (EST)

To truncate a tar file you need Hex Editor in combination with dd tool.

Something like hexdump -C data_00010.tar | more -- To get Hex Location of time stamp

and then dd if=./data_00010.tar of=./data_00010_copy.tar bs=<location> count=1 -- To truncate. Note that "Location" is decimal value so you might need to convert Hex to Decimal to use this command.


For example
$ tar -tvf data_00010.tar | tail -4f
-rwxrwxrwx 0/0 20 2011-11-23 17:24 commit-1322087067980.sh
-rwxrwxrwx 0/0 0 2011-11-23 18:20 1322090435044/398a250e-96f7-4029-a083-4c78f426e56d.n
-rwxrwxrwx 0/0 203 2011-11-23 18:20 1322090435044/7efe2484-5818-4638-8ccb-08ad9c2fe4c0.na
-rwxrwxrwx 0/0 20 2011-11-23 18:20 commit-1322090435044.sh

$ hexdump -C data_00010.tar | tail -100f

03ac3400 63 6f 6d 6d 69 74 2d 31 33 32 32 30 38 37 30 36 |commit-132208706|
03ac3410 37 39 38 30 2e 73 68 00 00 00 00 00 00 00 00 00 |7980.sh.........|
03ac3420 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
03ac3460 00 00 00 00 20 20 20 37 37 37 20 00 20 20 20 20 |.... 777 . |
03ac3470 20 30 20 00 20 20 20 20 20 30 20 00 20 20 20 20 | 0 . 0 . |
03ac3480 20 20 20 20 20 32 34 20 31 31 36 36 33 32 37 31 | 24 11663271|
03ac3490 32 33 33 20 20 20 20 37 32 32 32 20 30 00 00 00 |233 7222 0...|
03ac34a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
03ac3600 6d 76 20 31 33 32 32 30 38 37 30 36 37 39 38 30 |mv 1322087067980|
03ac3610 2f 2a 20 2e 00 00 00 00 00 00 00 00 00 00 00 00 |/* .............|
03ac3620 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
03ac3800 31 33 32 32 30 39 30 34 33 35 30 34 34 2f 33 39 |1322090435044/39|
03ac3810 38 61 32 35 30 65 2d 39 36 66 37 2d 34 30 32 39 |8a250e-96f7-4029|
03ac3820 2d 61 30 38 33 2d 34 63 37 38 66 34 32 36 65 35 |-a083-4c78f426e5|
03ac3830 36 64 2e 6e 00 00 00 00 00 00 00 00 00 00 00 00 |6d.n............|
03ac3840 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
03ac3860 00 00 00 00 20 20 20 37 37 37 20 00 20 20 20 20 |.... 777 . |
03ac3870 20 30 20 00 20 20 20 20 20 30 20 00 20 20 20 20 | 0 . 0 . |
03ac3880 20 20 20 20 20 20 30 20 31 31 36 36 33 32 37 37 | 0 11663277|
03ac3890 37 30 33 20 20 20 31 32 31 31 30 20 30 00 00 00 |703 12110 0...|
03ac38a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

Now in above example suppose you want to truncate on 1322090435044/398a250e-96f7-4029-a083-4c78f426e56d.n Now see corresponding entry in Hex output

03ac3800 31 33 32 32 30 39 30 34 33 35 30 34 34 2f 33 39 |1322090435044/39|
03ac3810 38 61 32 35 30 65 2d 39 36 66 37 2d 34 30 32 39 |8a250e-96f7-4029|
03ac3820 2d 61 30 38 33 2d 34 63 37 38 66 34 32 36 65 35 |-a083-4c78f426e5|
03ac3830 36 64 2e 6e 00 00 00 00 00 00 00 00 00 00 00 00 |6d.n............|
03ac3840 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

Then get first value of hex that is "03ac3800" in this case and convert it to decimal using any tool (http://www.binaryhexconverter.com/hex-to-decimal-converter). Decimal value would be "61618176". Then use following command to truncate

dd if=./data_00010.tar of=./data_00010_copy.tar bs=61618176 count=1

Now if you will examine both file (See data tar file is truncated (data_00010_copy.tar) and original file is under data_00010.tar)

$ tar -tvf data_00010.tar | tail -6f
-rwxrwxrwx 0/0 20 2011-11-23 16:09 commit-1322082572919.sh
-rwxrwxrwx 0/0 168 2011-11-23 17:24 1322087067980/fb042de1-85f4-4ca3-b4ee-e083ea79753c.na
-rwxrwxrwx 0/0 20 2011-11-23 17:24 commit-1322087067980.sh
-rwxrwxrwx 0/0 0 2011-11-23 18:20 1322090435044/398a250e-96f7-4029-a083-4c78f426e56d.n
-rwxrwxrwx 0/0 203 2011-11-23 18:20 1322090435044/7efe2484-5818-4638-8ccb-08ad9c2fe4c0.na
-rwxrwxrwx 0/0 20 2011-11-23 18:20 commit-1322090435044.sh
$ tar -tvf data_00010_copy.tar | tail -6f
-rwxrwxrwx 0/0 20 2011-11-23 12:14 commit-1322068449556.sh
-rwxrwxrwx 0/0 0 2011-11-23 16:09 1322082572919/ef65dcfa-000a-4ede-ab3d-4844ee66d242.n
-rwxrwxrwx 0/0 92 2011-11-23 16:09 1322082572919/97daadf7-770b-4005-a1f1-86040b32b9b5.na
-rwxrwxrwx 0/0 20 2011-11-23 16:09 commit-1322082572919.sh
-rwxrwxrwx 0/0 168 2011-11-23 17:24 1322087067980/fb042de1-85f4-4ca3-b4ee-e083ea79753c.na
-rwxrwxrwx 0/0 20 2011-11-23 17:24 commit-1322087067980.sh


1) http://www.binaryhexconverter.com/hex-to-decimal-converter
2) http://www.novell.com/communities/node/6419/making-sense-hexdump
3) http://en.wikipedia.org/wiki/Dd_(Unix)

Thanks Thomas Mueller From Adobe for providing this information.

Caution: This command is very dangerous and lead to data loss. Please use it with "A lot" of caution.

How to write CQ / WEM init script for Linux

Use case : You want to start and stop CQ on system start and stop

Solution: Here is script that you can use. Put this script in /etc/init.d


1. Create a file /etc/init.d/cq5 on your server and copy the contents from below script.
2. edit the file and make the following changes
update the "pidfile:" config to match your cq instance
update the "CQ5_ROOT" and "CQ5_USER" variables to match your cq instance
3. Run this command:
chmod 755 /etc/init.d/cq5
4. Run this command to add the config to redhat startup and shutdown:
chkconfig --add cq5

#!/bin/bash
#
# /etc/rc.d/init.d/cq5
#
#
#
#
# # of the file to the end of the tags section must begin with a #
# character. After the tags section, there should be a blank line.
# This keeps normal comments in the rest of the file from being
# mistaken for tags, should they happen to fit the pattern.>
#
# chkconfig: 35 85 15
# description: This service manages the Adobe WEM Content Management java process.
# processname: cq5
# pidfile: /crx-quickstart/server/logs/cq.pid

# Source function library.
. /etc/rc.d/init.d/functions

CQ5_ROOT=/opt/cq5
CQ5_USER=cq5

########
SERVER=${CQ5_ROOT}/crx-quickstart/server
START=${SERVER}/start
STOP=${SERVER}/stop
STATUS="${SERVER}/serverctl status"

case "$1" in
start)
echo -n "Starting cq5 services: "
su - ${CQ5_USER} ${START}
touch /var/lock/subsys/cq5
;;
stop)
echo -n "Shutting down cq5 services: "
su - ${CQ5_USER} ${STOP}
rm -f /var/lock/subsys/cq5
;;
status)
su - ${CQ5_USER} ${STATUS}
;;
restart)
su - ${CQ5_USER} ${STOP}
su - ${CQ5_USER} ${START}
;;
reload)
;;
*)
echo "Usage: cq5 {start|stop|status|reload}"
exit 1
;;
esac




For CQ5.5



#!/bin/bash
#
# /etc/r.d/init.d/cq5author
#
# of the file to the end of the tags section must begin with a #
# character. After the tags section, there should be a blank line.
# This keeps normal comments in the rest of the file from being
# mistaken for tags, should they happen to fit the pattern.>
#
# chkconfig: 35 85 15
# description: This service manages the Adobe WEM Content Management java process.
# processname: cq5author
# pidfile: /crx-quickstart/conf/cq.pid

# Source function library.
. /etc/rc.d/init.d/functions

CQ5_ROOT=/opt/Adobe/cq5_5/author
CQ5_USER=root

########
SERVER=${CQ5_ROOT}/crx-quickstart
START=${SERVER}/bin/start
STOP=${SERVER}/bin/stop
STATUS="${SERVER}/bin/status"

case "$1" in
start)
echo -n "Starting cq5 services: "
su - ${CQ5_USER} ${START}
touch /var/lock/subsys/cq5
;;
stop)
echo -n "Shutting down cq5 services: "
su - ${CQ5_USER} ${STOP}
rm -f /var/lock/subsys/cq5
;;
status)
su - ${CQ5_USER} ${STATUS}
;;
restart)
su - ${CQ5_USER} ${STOP}
su - ${CQ5_USER} ${START}
;;
reload)
;;
*)
echo "Usage: cq5 {start|stop|status|reload}"
exit 1
;;
esac

Thanks Andrew Khoury and Daniel Gordon from Adobe for providing this information.

How to mount DataStore in different directory in CQ / WEM

Use Case : You have huge datastore and want to mount it on different directory Or Online backup takes a lot of time including datastore and you want to do separate backup for datastore

Solution : Here is process for how to separate Datastore and then do separate online backup

Assuming your repository is under /mnt/crx/repository and you want to move your datastore to /mnt/crx/datastore

1. stop the crx instance
2. mv /mnt/crx/repository/shared/repository/datastore /mnt/crx/
3. Then modify repository.xml by adding the new path configuration to the DataStore element.

Before:
<DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
<param name="minRecordLength" value="4096"/>
</DataStore>

After:
<DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
<param name="path" value="/mnt/crx/datastore"/>
<param name="minRecordLength" value="4096"/>
</DataStore>

4. Start CRX

After doing this then you can safely run separate backups on the datastore while the system is running without affecting performance very much.

Following our example, you could use this command to backup the datstore:
rsync --av --ignore-existing /mnt/crx/datastore /mnt/backup/datastore

Information about rsync can be found here http://ss64.com/bash/rsync.html

How to create package based on Xpath in CQ5 / WEM

Use Case : Some time you want to create package based on Xpath. CQ5 package manager does not have ability to create package based on Xpath.

Solution :

You can use following package to achieve this

FOR CQ5.3 and CQ5.4



FOR CQ5.5



FOR CQ5.6 (Also fixes some other issues)



FOR CQ6

Above package might not work for AEM 6 onward because of this feature You might have to do disable POST from felix console for CSRF token. Add it back after running this tool.

1) Download and Install package using package manager
2) go to <host>:<port>/apps/tools/components/createPackage/run.html
3) Give your Xpath in xpath value
4) You can also add comma separate exclude path that you don't want to add to package.
5) Click on Create config package
6) Now Download the package and also be saved under /etc/packages/CQSupportTool

For example if you have to create package of all ACL to migrate from one CQ instance to another you can use xpath query for package as //element(*,rep:ACL)

Please note that, this package is for test purpose only. Feel free to modify it based on your need.

Known Issue Exception when "/" root is given (I will fix that as soon as get some time).

How to fix com.day.crx.cluster.LoginException: Slave has same identity as master Error

Symptom: You get Caused by: com.day.crx.cluster.LoginException: Slave has same identity as master error and CQ does not start up. This usually happen in CQ5.3 or upgraded CQ5.4 instance.

Solution :

-Stop instance
-Delete all files named listener.properties in

* crx-quickstart/repository/shared/journal/control
* crx-quickstart/repository/shared/version/control
* crx-quickstart/repository/shared/workspaces//control

- Delete file crx-quickstart/repository/cluster_node.id
- Delete all stale .lock files (if existing)
- Delete all stale lock.properties files (if existing)
- Start CQ5.3 instance

OR
1) Use following following command to remove all listener.properties files
find ./crx-quickstart/repository/shared -name listener.properties | xargs rm -f

2) - Delete file crx-quickstart/repository/cluster_node.id

3) Delete all .lock files (if existing) using
find ./crx-quickstart/repository -name "\.lock" -exec rm '{}' \; -print

How to fix File Not found issue in CRX

Caution : Most of time it works but not guaranteed

Assumption : You are using CRX and latest HF is not installed. Otherwise need to figure out why you had this issue.

Symptom : You get following error in log

Caused by: org.apache.jackrabbit.core.state.ItemStateException: Failed to read bundle: <SOME -- UUID> : java.io.IOException: File not found: 263
at com.day.crx.persistence.tar.TarPersistenceManager.getInputStream(TarPersistenceManager.java:1195)
at com.day.crx.persistence.tar.TarPersistenceManager.loadBundle(TarPersistenceManager.java:334)
at com.day.crx.persistence.tar.TarPersistenceManager.loadBundle(TarPersistenceManager.java:311)
at org.apache.jackrabbit.core.persistence.bundle.AbstractBundlePersistenceManager.getBundle(AbstractBundlePersistenceManager.java:654)
at org.apache.jackrabbit.core.persistence.bundle.AbstractBundlePersistenceManager.load(AbstractBundlePersistenceManager.java:400)
at org.apache.jackrabbit.core.state.SharedItemStateManager.loadItemState(SharedItemStateManager.java:1819)
at org.apache.jackrabbit.core.state.SharedItemStateManager.getNonVirtualItemState(SharedItemStateManager.java:1739)
at org.apache.jackrabbit.core.state.SharedItemStateManager.getItemState(SharedItemStateManager.java:261)
at org.apache.jackrabbit.core.state.LocalItemStateManager.getNodeState(LocalItemStateManager.java:107)
at org.apache.jackrabbit.core.state.LocalItemStateManager.getItemState(LocalItemStateManager.java:172)
at org.apache.jackrabbit.core.state.XAItemStateManager.getItemState(XAItemStateManager.java:260)
at org.apache.jackrabbit.core.state.SessionItemStateManager.getItemState(SessionItemStateManager.java:161)
at org.apache.jackrabbit.core.ItemManager.getItemData(ItemManager.java:370)
... 34 more
Caused by: java.io.IOException: File not found: 263
at com.day.crx.persistence.tar.TarSet.getInputStream(TarSet.java:731)
at com.day.crx.persistence.tar.TarSet.getInputStream(TarSet.java:724)
at com.day.crx.persistence.tar.ClusterTarSet.getInputStream(ClusterTarSet.java:502)
at com.day.crx.persistence.tar.TarPersistenceManager.getInputStream(TarPersistenceManager.java:1191)
... 46 more



Solution

1) stop the instance
2) Make sure that it stopped (You can run ps -ef | grep java command to check that)
3) delete [take backup] all index_*.tar files found in /crx-quickstart/repository/workspaces/crx.default/copy (If it is CQ5.3 or upgraded instance)
4) Grep the error.log to find the missing tar file numbers:
grep "^java.io.IOException: File not found:" error.log* | awk '{ print $5 }' | sort –u
5) from your nightly backups Recover and place missing tar file "Data*.tar" at repository/workspace/crx.default/
6) Set the file permissions back to rw as the system sets all data tar files to read only after a file is missing. chmod 755 *.tar.
7) Start CRX / CQ with the following system property (in start up script):
java -Dcom.day.crx.persistence.tar.IndexMergeDelay=0
8) Make sure that you have latest CRX Hotfix installed


If above issue happen in clustered environment, You can recover it from working cluster instance. See How to recover from Out of sync node in cluster

Please do not attempt to do this directly on production instance without taking help of Adobe Support

How to remove lock files in CQ / WEM

Problem :

javax.jcr.RepositoryException: The repository home /crx-quickstart/repository appears to be in use since the file named .lock is locked by another process.
at org.apache.jackrabbit.core.util.RepositoryLock.tryLock(RepositoryLock.java:166)
at org.apache.jackrabbit.core.util.RepositoryLock.acquire(RepositoryLock.java:138)
at org.apache.jackrabbit.core.RepositoryImpl.(RepositoryImpl.java:276)
at com.day.crx.core.CRXRepositoryImpl.(CRXRepositoryImpl.java:233)
at com.day.crx.core.CRXRepositoryImpl.(CRXRepositoryImpl.java:288)
at com.day.crx.core.CRXRepositoryImpl.create(CRXRepositoryImpl.java:193)
at com.day.crx.j2ee.CRXRepositoryStartupServlet.createRepository(CRXRepositoryStartupServlet.java:173)

Reason why above happen, If you forcefully shut down CQ sometimes .lock files does not get deleted.

Solution :

You can use following command in Linux/ Unix platform to delete all lock files

find <Path to crx-quickstart>/repository -name "\.lock" -exec rm '{}' \; -print

How to use SVN with VLT in CQ / WEM

Use Case : You can not do all the operation using svn with CRXDE or CRXDE light. Also if you are using some other development tool, You can use File system to perform SVN operations.

Pre requisite : You have svn and vlt already set up
For svn set up use svn product guideline
for vlt set up refer http://dev.day.com/docs/en/crx/current/how_to/how_to_use_the_vlttool.html

Solution :

Initial Set up :
$ cd /projects
$ svn mkdir http://svn.server.com/repos/myproject
$ svn co http://svn.server.com/repos/myproject
$ vlt export -v http://localhost:4502/crx /apps/geometrixx geometrixx
$ cd geometrixx/
$ svn add META-INF/ jcr_root/
$ svn ci

Changes made in File System and update to local repository (JCR)  If you are using vlt sync then you don't have to do this:

$ cd /projects/geometrixx
$ vlt -v import http://localhost:4502/crx . /
Or
$ vlt add <File you added>
$ vlt ci
Or
$ vlt st
$ vlt ci

Changes made in SVN and update to File system :
$ cd /projects/geometrixx
$ svn st
M META-INF/vault/properties.xml
M jcr_root/apps/geometrixx/components/contentpage/.content.xml
$ svn up
If you are using vlt sync then you don't have to use this option
$ vlt ci


Changes made in repository and update it back to file system (With vlt sync you don't have to use this)
1) go to: content/jcr_root/<Path>
2) vlt up --force


Changes made in a file in local file system and update repository:

For example you are changing one file

1) go to: .../jcr_content/libs/foundation/components/text
2) edit something in text.jsp
3) see the modified files vlt st
4) see the changes: vlt diff text.jsp
5) commit the changes: vlt ci test.jsp
6) reload page containing a text component and see that it worked



So direction could be any thing (After initial set up, You don't have to do it when using vlt sync)

repo change ---> vlt st ---> vlt up --> svn st --> svn ci

svn change ----> svn st --> svn up --> vlt ci

file system change ---> vlt st ---> vlt ci And then svn st ----> svn ci

Now with vlt synch option you can synch between your eclipse project and crx. This helps not to use vlt ci every time you make file system changes and use vlt up when you make crxde changes.

More infirmation about vlt synch can be found here http://dev.day.com/docs/en/crx/current/how_to/how_to_use_the_vlttool.html#Using%20VLT%20sync%20for%20development

How to read an external file in CQ

Use Case: You want to read something from external file

Solution : There are multiple ways you can do this I will mentioned two of them here,

1) You can put them under /resources folder under /src/main for example if you want to read a file called "test.txt" then you can put file under /src/resources/<any-sub-foler>/<file-to-read>

Then use following code to read file
InputStream stream = getClass().getResourceAsStream("<file-to-read>");
in above case it would be InputStream stream = getClass().getResourceAsStream("/test.txt");

Good practice is to follow convention, For example if your source code is under /src/main/java/com/test/mytest/Test.java Then your resource should be in location /resource/com/test/myTest/<Your File>

Then you can use something like

InputStream in = Test.class.getResourceAsStream("<Your File>");

2) You can put file in repository (Any repository location is file) and use Node data as stream to read.

then use following code,
InputStream in = node.getNode("jcr:content").getProperty("jcr:data").getStream();

How to Upload File in CRX

File file = new File("/mypath/mydocument.pdf");

FileInputStream is = new FileInputStream(file);

mimeType = "application/octet-stream";

Node node = session.getNode(parentNode);

ValueFactory valueFactory = session.getValueFactory();

Binary contentValue = valueFactory.createBinary(is);

Node fileNode = node.addNode(fileName, "nt:file");

fileNode.addMixin("mix:referenceable");

Node resNode = fileNode.addNode("jcr:content", "nt:resource");

resNode.setProperty("jcr:mimeType", mimeType);

resNode.setProperty("jcr:data", contentValue);

Calendar lastModified = Calendar.getInstance();

lastModified.setTimeInMillis(lastModified.getTimeInMillis());

resNode.setProperty("jcr:lastModified", lastModified);

session.save();

return fileNode.getPath();

Using Sling FSResource Provider (To convert File system resource to jcr resource):

Please see what all different kind of resource provider is present

http://sling.apache.org/documentation/the-sling-engine/resources.html

FS resource provider essentially mapped your file system path with repository path. That mean you can adapt your file system resource to a sling resource object and can do fun stuff with it.

1) More information can be obtained here http://sling.apache.org/documentation/bundles/accessing-filesystem-resources-extensions-fsresource.html

2) FSResource provider is not present OOTB, so you have to first embed this in your project,

in your pom.xml do following (I hope you know how to do this, if not ping me :D),

<embedded>

<groupId>org.apache.sling</groupId>

<artifactId>org.apache.sling.fsresource</artifactId>

<target>/apps/<your app>/install</target>

</embedded>

And

<dependency>

<groupId>org.apache.sling</groupId>

<artifactId>org.apache.sling.fsresource</artifactId>

<version>1.1.3-SNAPSHOT</version>

</dependency>

3) create a osgi config with following setting 

Name of file: org.apache.sling.fsprovider.internal.FsResourceProvider-samples.xml

<?xml version="1.0" encoding="UTF-8"?>

<jcr:root xmlns:sling="http://sling.apache.org/jcr/sling/1.0" xmlns:jcr="http://www.jcp.org/jcr/1.0"

          jcr:primaryType="sling:OsgiConfig"

          provider.roots=“<some repository path, this can be any random path>"

          provider.file=“<Absolute path for file in file system>"

          provider.checkinterval="1000”/>

4) Then in your code you can do following,

import java.io.File;

import org.apache.felix.scr.annotations.Component;

import org.apache.felix.scr.annotations.Reference;

import org.apache.felix.scr.annotations.Service;

import org.apache.sling.api.resource.LoginException;

import org.apache.sling.api.resource.Resource;

import org.apache.sling.api.resource.ResourceProvider;

import org.apache.sling.api.resource.ResourceResolver;

import org.apache.sling.api.resource.ResourceResolverFactory;

import org.slf4j.Logger;

import org.slf4j.LoggerFactory;

@Component 

@Service

public class FileSystemFileTest implements FileSystemTest {

private static final Logger LOG = LoggerFactory.getLogger(FileSystemFileTest.class);

@Reference

ResourceResolverFactory resourceResolverFactory;

@Reference

ResourceProvider resourceProvider;

ResourceResolver resourceResolver = null;

@Override

public File getFile() {

try {

resourceResolver = resourceResolverFactory.getAdministrativeResourceResolver(null);

Resource resource =  resourceResolver.getResource("<some repository path from above>");

File file  = resource.adaptTo(File.class);

if(null!=file){

return file;

}

} catch (LoginException e) {

LOG.error(e.getMessage());

}finally{

if(null!=resourceResolver){

resourceResolver.close();

}

}

return null;

}

}

Note that in this case you need to have file in your file system. You will essentially use this approach if you want to get jcr resource view of your file system to do something with it. For example if your method can only accept resource object as argument.

How to set up User/Group/Profile reverse replication

Assumption: You are using CQ5.4

Use Case: You are creating users and group on publish and want to move them on author

Prerequisite: http://helpx.adobe.com/cq/kb/HowToUseReverseReplication.html

Solution:

Create two launcher on publish with run mode as publish which listen to rep:User and rep:Group on path /home/user and /home/group respectively and trigger reverse replication OR create one with rep:Authorizable and listen to /home
So it would be like

For User and Group Option 1

Event Type: Created
Nodetype: rep:User
Path: /home/users
Condition: cq:distribute!=
Workflow: /etc/workflow/models/reverse_replication
Run Mode: publish

AND

Event Type: Created
Nodetype: rep:Group
Path: /home/group
Condition: cq:distribute!=
Workflow: /etc/workflow/models/reverse_replication
Run Mode: publish

For User and Group Option 2

Event Type: Created
Nodetype: rep:Autherizable
Path: /home
Condition: cq:distribute!=
Workflow: /etc/workflow/models/reverse_replication
Run Mode: publish


For Profile

A. Code Changes
Change your code/form, that a profile update POST is adding
cq:distribute=true
source=authorname
as key=value into the /profile folder.

B. Setup on Publish Instance
Add Workflow Launcher Configuration:
Event Type: Modified
Nodetype: sling:Folder
Path: /home/users/(.*)/profile
Condition: cq:distribute!=
Workflow: /etc/workflow/models/reverse_replication
Run Mode: publish

C. Setup on Author Instance
Create Workflow Model:
One process step
Implementation: com.day.cq.wcm.workflow.process.ActivatePageProcess
Add Workflow Launcher Configuration:
Event Type: Modified
Nodetype: sling:Folder
Path: /home/users/(.*)/profile
Condition: source!=
Workflow: select your model created above
Run Mode: author


Important Note:
1) Above method will only work for created trigger. If you are modifying user profile on publish then you have to do same thing for modified trigger too.

2) If you have multiple publish instance then you have to make sure that all the publish instance are in sync. For that you need to again replicate those user back to other publish instance using launcher on author.
See an example how comment activation work. when user post a comment in publish instance a comment post servlet get called which add cq:distribute property to that node along with other properties -> this make node eligible for reverse replication -> publish instance put that node in outbox -> author reverse_publish agent poll data from publish and put it in /content/usergenerated -> Workflow launcher on Node Created /content/usergenerated of type cq:comment get called -> this calls workflow associated with it (Which is Comment Moderation workflow) -> workflow has logic to activate comment back to all publish instance based on whether it requires moderation or not (Simple replicator.replicate)


Thanks Cédric Hüsler from Adobe to provide this information

How CQ (Till CQ5.4) Starts when you click on quickstart jar file

1. the quickstart is unpacked to the crx-quickstart directory
2. then cqse is started
3. cqse starts the crx webapp
4. the crx webapp starts the repository
5. during repository startup, it checks the repository/install folder for content packages and installs them
6. then the launchpad webapp is started
7. felix starts the core bundles
8. one of the core bundles is the JCR installer, which searches all 'install' nodes in the repository and registers those bundles
9. felix starts the newly registered bundles
10. As a part of newly started bundles CQ apps load

Thanks Toby from Adobe for this information

How to make sure that Links are always valid on a page

Use Case: Some time link checker makes certain links as invalid (As it is not able to verify it).

Solution: You can add x-cq-linkchecker="valid" parameter in the <a> tag to make sure that links are always mark as valid by CQ. In this case link checker will check the link but will mark it valid.

You can optionally use x-cq-linkchecker="skip" in the <a> as well. In this case link checker will not even check for validity for link.

There are other options too to make all external link as valid,

Option 2: You can disable external link checker entirely by going to felix console



Option 3: Additionally you can add override pattern to disable link checking for certain domain



Example for override pattern: for http://www.day.com you will write ^http://www\.day\.com/

Note:
Following error with respect to external link checker,
com.day.cq.rewriter.linkchecker.impl.LinkInfoStorageImpl No more external links allowed for host <Host Name> Maximum of 128 reached. means,
External link checker only checks first 128 link per host basis, And you can find those entry under /var/linkchecker/<protocol>. Unfortunately this is not configurable yet.

How to flush Cache using Curl Command in Adobe CQ / AEM

Use Case Flush cache for a path in regular interval of time

Solution You can use following command to flush cache


curl -H "CQ-Action:Flush" -H "CQ-Handle: /content/geometrixx/en/toolbar" -H "CQ-Path:/content/geometrixx/en/toolbar" -H "Content-Length: 0" -H "Content-Type: application/octet-stream" http://dispatcher-server-hostname:port/dispatcher/invalidate.cache

In Newer version of dispatcher try this

curl -H "CQ-Action: DELETE" -H "CQ-Handle:/content/geometrixx/en/toolbar" -H "CQ-Path:/content/geometrixx/en/toolbar" -H "Content-Length: 0" -H "Content-Type: application/octet-stream" http://dispatcher-server-hostname:port/dispatcher/invalidate.cache

Expected Response: <H1>OK</H1>

If you see error like
[Thu May 30 09:23:53 2013] [D] [91650(140735211479424)] Found farm <Your FARM>
[Thu May 30 09:23:53 2013] [D] [91650(140735211479424)] checking [/dispatcher/invalidate.cache]
[Thu May 30 09:23:53 2013] [W] [91650(140735211479424)] Flushing rejected from <Something>
That mean you are not in allowed list of client to flush cache. In that case please allow your IP (<SOMETHING>) in above case in allowedClient section of dispatcher.any

 /allowedClients
        {
         /0000
          {
          /glob "*"
          /type "deny"
          }
        /0001
          {
          /glob "<Something>"
          /type "allow"
          }
... More allowed
        }
      }

Important Note: Make sure that from outside (DMZ may be) no one should be able to run this command. This command should be allowed to run only from some trusted boxes

Thanks Andrew Khoury from Adobe for this information

How to rotate TarJournal in Shared Nothing Clustering

Use Case: Your tar Journal is growing and consuming a lot of space.

Solution: If you are using shared clustered Please refer http://dev.day.com/content/kb/home/Crx/Troubleshooting/JournalTooMuchDiskSpace.html

For shared nothing clustering you need following configuration (If you want rotation after every 24 Hour) in repository.xml

<Journal class="com.day.crx.persistence.tar.TarJournal">
<param name="maxFileSize" value="104857600" />
<param name="maximumAge" value="PT24H" />
</Journal>

Please note that Age specified as duration in ISO 8601 or plain format. Journal files that are older than the configured age are automatically deleted. The default is "P1M", which means files older than one month are deleted.

More detail can be found here http://en.wikipedia.org/wiki/ISO_8601

Important Note : If you do above configuration then make sure that other instance in the cluster is not down for more than specified time in "maximumAge", Otherwise cluster will get Out Of sync because of rotation of tarJournal.

How to find all the pages modified or activated after certain time

Use Case: You want to create package of all the nodes modified after certain date and time. This can be use ful in Migration Projects. If you have problem with one of the activation agent and you want to activate pages / Asset to that agent based on time.

Solution: You can use following packages to achieve this use case



You can also refer code form Here

1) Install attached package using package manager
2) Go to http://<host>:<port>/apps/tools/components/listPage.html
3) Select Date and time
4) Then click on submit
5) You can click on create package of modified pages to create package


Feel free to change the code based on your requirement

How to clone publish server in CQ

There are multiple way to clone the publish server

With downtime::

1) Create new replication agent for other server.
2) Stop publish server you want to clone
3) take backup of crx-quickstart folder
4) put it on other server
5) Start publish 1 and then publish 2
6) once both the server started pending replication queue will get clear

Without down time

1) Create new replication agent for other server
2) Take online back up of publish 1
3) put online backup on server 2
4) restore online backup
5) start publish 2
6) All the queue pending at publish2 replication agent will get clear.

After the clone change disaptcher.any to add extra renderer.

How to get Client IP through dispatcher in CQ

Try following code to get client IP address.

// headers in order of trust, most trusted at top

Then in your JSP you can just do

<%=getClientIpAddr(slingRequest)%>

Note: Make sure that all headers are allowed from dispatcher

At dispatcher side make sure that you are allowing all the headers with this configuration in dispatcher.any

/clientheaders
      {
      "*"
      }

How to remove version history in CQ / WEM

Problem:
Over time the Version store /jcr:system/jcr:versionStorage can grow to a considerable size. You see that when:
lots of tar files in (CQ5.3):
/crx-quickstart/repository/version/copy
/crx-quickstart/repository/shared/version
large lucene index on:
/crx-quickstart/repository/repository/index

This is also helpful when you are upgrading from CQ5.2X to latest version (As version purging was not enabled in previous version)

Solution: There are couple of solution to this,

You could enable automatic version purging. See more information here

You can also remove version using code mention here

How to Set up SSL for local in CQ / WEM / Dispatcher

Use case For testing

Step 1: Generate Key Store

Use Keytool to generate keystore

You can also confirm if you have keytool in your system by using command keytool --help

Once you have keytool installed you can following command to generate ketstore

From command line navigate to /crx-quickstart/server/etc
Use command "keytool -genkey -keystore mykey -alias myalias -keyalg rsa"

Set up SSL till CQ5.4 (Using server.xml)

use following entry in server.xml

<container>
<listener>
<!-- You already have one entry here don't modify it-->
</listener>
<!--Entry for new SSL Listener-->
<listener>
<bind-port>443</bind-port>
<ssl>
<protocol>SSL</protocol>
<key-store>
<name>etc/mykey</name>
<passphrase><Password you have given while creating certificate></passphrase>
</key-store>
<key>
<alias>myalias</alias>
<password><Password you have given while creating certificate></password>
</key>
</ssl>
</listener>
<!--End of new entry for SSL-->
</container>


You can also check /crx-quickstart/server/etc/SSL_HowTo.txt to see how configuration can be done.

NOTE:

Once you have SSL set up check logs/server.log to make sure that server is started on secure port.

If you get Error like,

*ERROR* servletengine: Unable to start https listener on address 127.0.0.1, port 443: Permission denied
That means you need to start CQ as root user.

Set up SSL in CQ5.5

In CQ5.5 CQSE is deployed as a bundle and you can configure SSL using Felix configuration, Please see screen shot of how to do that. All parameter is self explanatory



This is actual configuration




Note
1. You can put certificate file at any location you want. Only absolute path is required.
2. There is no way to configure multiple port you can listen to now.

Set up SSL on apache (If your SSL terminate at apache)

Assuming that you are using Apache web server,

Click here to see how to generate certificate and key file

If you already have cert and password then you can use following command to generate key

openssl rsa -in <Your Key>.key -out <Key with Password>.new.key

Then go to /conf/httpd.conf and add following entry

Listen 443
<VirtualHost *:80>
    ServerName wemblog.com
    ServerAlias wemblog*.com

    RewriteEngine on
    #Rewrite all request to https
    RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [L,R=301]
</VirtualHost>

NameVirtualHost *:443
Listen 443
<VirtualHost *:443>
    ServerName wemblog.com
    ServerAlias wemblog*.com
    SSLEngine on
    SSLCertificateFile <cert path>.cert
    SSLCertificateKeyFile <key path>.key

    RewriteEngine on
  
    ProxyPreserveHost On
    ProxyPass / http://localhost:4502/
    ProxyPassreverse / http://localhost:4502

    #set header for SSL
    Header add X-Forwarded-Proto "https"
    <LocationMatch "/(content|apps|etc).*">
        RequestHeader set X-Forwarded-Proto "https"
    </LocationMatch>

</VirtualHost>

To be honest you will get ton of information about How to set up SSL on Apache on Google.


Note: If you just have to use https (force https in CQ) and not let author to use http, There are two options,

1) You can configure dispatcher rewrite rule to redirect all http request to https.
2) If you are not using dispatcher, you can write rewrite rule under /etc/map to redirect all request to https port. Here is example

Please check https://cwiki.apache.org/SLING/flexible-resource-resolution.html


You have to do something like this

/etc/map
+-- http
+-- localhost.4502
+-- sling:redirect = "https://localhost:<your secure port>"

How to set up debug mode for Class/Package in CQ / CRX (Till CQ5.4)

Use Case You have problem with certain modules in CQ/CRX and you want to debug it

Setting Debug mode in CQ

For example if I have to set debug for package com.day.cq.dam.core

[1] Log into Felix Console: http://<host>:<port>/system/console/configMgr
[2] From "Factory Configurations", create "Apache Sling Logging Writer Configuration"
[2.1] Set value of "Log File" to "../logs/dam.log"
[2.2] Click on "Save"
[3] From "Factory Configurations", create "Apache Sling Logging Logger Configuration"
[3.1] Set value of "Log Level" to "Debug"
[3.2] Set value of "Log File" to "../logs/dam.log"
[3.3] Add "Logger" => com.day.cq.dam.core
[3.4] Click on "Save"

Now all debug information for package com.day.cq.dam.core will be redirected to /crx-quickstart/logs/dam.log

Click Here to see how to Rotate custom logs in CQ.


Note: Note that OOTB logger information will be redirected

For example everything with

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
private final Logger logger = LoggerFactory.getLogger(getClass());
and then logger.debug("Your message");


Setting debug mode for CRX

Suppose you want to enable logging for com.day.crx.security.ldap package.

1. Open the log4j.xml file (located in crx-quickstart/server/runtime/0/_crx/WEB-INF/) and add another appender:

<!-- ldap.log -->
<appender name="ldap" class="org.apache.log4j.FileAppender">
<param name="File" value="crx-quickstart/logs/crx/ldap.log"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{dd.MM.yyyy HH:mm:ss} *%-5p* %m%n"/>
</layout>
</appender>


2. Add another logger:

<logger name="com.day.crx.security.ldap">
<level value="debug"/>
<appender-ref ref="ldap"/>
</logger>


NOTE

Optionally, you can use "com.day.crx.security" instead of "com.day.crx.security.ldap" to see all CRX security-related debug log messages in the logs/crx/ldap.log file.

3. Restart the server.

4. Try logging into CRX directly with your LDAP account.

5. Examine the ldap.log and error.log files from CRX to debug for errors.

NOTE

If you are troubleshooting LDAP for a CQ instance, first try logging CRX at http://<host>:<port>/crx. Second, try logging into CQ at http://<host>:<port>. Then examine the ldap.log and error.log files from CRX and CQ for errors.

For CQ5.5 there is no log4j.xml. All the CRX logger needs to configured using sling config. So If you want to debug LDAP in CQ5.5, You have to use sling logging logger configuration


Just as a side note you can start your CRXDE in debug mode by adding "-d" in the end of start option. For example to start CRXDE in debug mode you can chose java -jar <CQ jar file name> -d

How to add ACL to a node

Use case: Some time you want to associate ACL on a node based on certain events (For example page creation under certain path).

Solution: You can use this code to associate ACL with a node.

How to create pages using curl command in CQ / WEM

Use Case For testing (If you are trying to capture certain events or testing something in MSM)

Solution There are various way of creating pages using curl command in CQ Some of them are as follows

1) curl -u admin:admin –F cmd="createPage" -F label="" -F parentPath="/content/geometrixx/en/company" -F template="/apps/geometrixx/templates/contentpage" -F title="new page" http://localhost:4502/bin/wcmcommand

2) curl -u admin:admin -F "jcr:primaryType=cq:Page" -F "jcr:content/jcr:primaryType=cq:PageContent" -F "jcr:content/jcr:title=New Page" -F "jcr:content/sling:resourceType=geometrixx/components/contentpage" http://localhost:4502/content/geometrixx/en/page

3) curl --data jcr:primaryType=cq:page --user admin:admin http://<host>:<port>/content/geometrixx/en/toolbar/test3

And then

curl --data jcr:primaryType=PageContent --user admin:admin http://<host>:<port>/content/geometrixx/en/toolbar/test3/jcr:content


You can also use JAVA code to create pages in CQ.

Sling Content Manipulation POST API detail can be found here http://sling.apache.org/site/manipulating-content-the-slingpostservlet-servletspost.html

Thanks Henry Seginor, Paul from Adobe for providing this information

How to use vlt tool to copy data from one CRX to other CRX

Use Case Some time you want to migrate repository from one instance to another instance (Because of non repairable corruption in current repository). If your repository size is big, Package manager is not a viable option (As you end up creating a lot of packages)

Assumption You have basic knowledge about CQ structure and CRX. You are using crx2.2 or higher. Path of copy could differ if you are using older version of CRX.

Here are different options:

1) Use package manager

• Pros: Very simple to create and do not require command like knowledge
• Cons: Not suitable for Big packages. Slow some time

2) Use VLT (We are explaining how to use this in this article)

•  Pros: Faster than package manager. Do not consume package space like package manager.
• Cons: Slow I/O

3) Use Tool based on VLT like Recap http://adamcin.net/net.adamcin.recap/

• Pros: VLT rcp UI easy to see status and administer. Do not require command like knowledge.
• Cons: Slow I/O because it uses VLT under the hood.

4) Use Tool based on other transfer protocol like Grabit  https://github.com/TWCable/grabbit

• Pros: Faster than VLT
• Cons: Not Adobe supported. Need more dependency to set up.

Step 1: Make sure that you have vlt set up properly. More detail can be found here. In the end you can use vlt --version command to check if vlt is installed.

Step 2: Use following command to Migrate data between CRX

vlt rcp -r http://<login>:<password>@<source-host>:<port>/crx/-/jcr:root/<Source-Path> http://<login>:<password>@<destination-host>:<port>/crx/-/jcr:root/

Note

1) While you are migrating data from one CQ to another CQ, make sure that you stop launchpad application from <host>:<port>/admin (This will make sure that unnecessary workflows are not getting triggered during migration). If this is not possible make sure that you disable DAM related workflow by going to http://<host>:<port>/libs/cq/workflow/content/console.html and then launcher. Don't forget to re enable them after the copy and make sure that no DAM activities are going on during that time on target instance.

2) Note that copying user and groups does not mean all ACL are also copied. ACL are stored under /content node. In general you need to migrate following stuff (Plus if you have some thing else custom)

-- /content/<Your-site>

-- /apps/<Your-application>

-- /var/dam/<your-asset>

-- /content/dam/<your-asset>

-- /etc/design/<your-design>

-- /etc/tags/<your-tags>

-- /etc/workflow/<your-custom-stuff>

-- /etc/replication/<your-custom-replication>

Important Note: At any point of time (Even after Migration). You have to make sure that /content/dam and /var/dam are sync. After migration go to http://<host>:<port>/etc/dam/healthchecker.html and make sure that they are in sync by clicking check binaries (List entries missing in /var/dam) and check Asset (List entries missing in /content/dam). You might also do small test before you do big migration to make sure that all renditions are getting migrated fine.

If you want to process all asset again then just migrate /var/dam in batches (As Asset synchronization using workflow could be expensive) and enable launchpad or all workflows. In order to migrate Asset in baches you can use sleep between two rsync. Make sure that you monitor logs of target system for any OOM error or any issue.

You can have script to do above with all the path (So that you can re use them in future)

If do not have Login can download Recap from Here

How to read Logs from Browser in CQ

Use Case: Some time for whatever reason, You don't have an access to physical server thus can not read logs in real time.

Solution : Please follow steps below,

1) Download log reader war file from here

2) Go to http://<host>:<port>/admin

3) Add content as /logreader and upload war file you saved.

4) Once uploaded, You can go to http://<host>:<port>/logreader to read the log.

For CQ5.5 you can use system configuration to read on browser

For > CQ5.6 you can also go to HOST:PORT/system/console/status-slinglogs to view log files.

Caution:: Please don't keep this file in production (As it will consume resources). If your job is done remove it.

Thanks Sebastian Hoogenberk from Adobe for providing this information.

How to reindex Large Repository in CQ or CRX

Note: Please test this before using.

Re-indexing a large repository may take a lot of time and cause downtime. The below procedure allows to keep the production repository operational and do the re-indexing on a backup. Downtime of the production repository is limited to the time it takes to index changes that were done on the production system while the re-indexing was performed.

• Stop production repository
• Create a copy of the repository to a separate machine
• Start production repository again
• On the repository copy:
  • delete the index
  • start repository, which will start the indexing process
  • once repository is up and running (indexing completed), stop it again
• stop production repository
• replace crx-quickstart/repository/workspaces/crx.default/index with the one from the copy
• replace crx-quickstart/repository/revision.log with the one from the copy
• rename crx-quickstart/repository/cluster_node.id to crx-quickstart/repository/cluster_node.id.orig
• start production repository

At this point the repository will apply the changes to the index that were done since the copy was made.

• Once the repository is online, shut it down again.
• Delete crx-quickstart/repository/cluster_node.id
• Rename crx-quickstart/repository/cluster_node.id.orig to cluster_node.id

CRX Clustering

All the information about CRX clustering can be found here, Please feel free to add comment.

How to remove eclipse based CRXDE cache

Why: Some time there is class path issues with CRXDE. You can clear CRXDE cache by removing ".crxde" under your home directory. For example in unix system if my username is test then you can find ".crxde" under /Users/test.

Note that .crxde is hidden folder.

You can also make jar file to be available for your CRXDE under class path by adding that jar file to /etc/crxde/profiles/default/libs
(You can use any webdev tool to do that). By default any thing under /install folder are available in class path.

How to remove repository inconsistency using redo.log in CQ5 / WEM

You can use redo.log to remove repository inconsistency (Instead of rebuilding whole index).

1. Find all the nodes {Please see below for that}
2. stop the instance
3. check the directory crx-quickstart/repository/workspaces/crx.default/index
4. there should be a file named 'indexes_X', e.g. indexes_1y
5. rename the redo_X.log so that it matches the suffix from above, e.g. redo_1y.log
6. place the redo-log in the crx-quickstart/repository/workspaces/crx.default/index folder
restart

Redo-log file will have entries like this

1 DEL 003171fe-e2e8-457b-a3af-f74eed12c1b9
1 DEL d4748598-374a-43a3-95a5-f53c94df8c7d
1 DEL fd8747eb-38c5-4256-b416-ad37b804a9e9
1 COM



How to find all inconsistent node
[1] Copy This file to your local system
[2] Open FindCorruption.java
[3] Change following variables to your local file system path
INPUT_FILE_NAME (This is location of your log file)
OUTPUT_FILE_NAME_SEARCH (Location where you want output to save)
OUTPUT_FILE_NAME_TAR
OUTPUT_FILE_NAME_MISSING_CHILD
OUTPUT_FILE_NAME_ORPHAN_CHILD
[4] Save and use javac FindCorruption.java
[5] Then run java FindCorruption
[6] For search index you can use it as redo.log
[7] for tar pm inconsistency you can use selective UUID (With consistencyCheckUUIDs param in repository.xml) to fix inconsistency. More information can be found here
[8] For missing child or orphan node you need to use console.sh (If it is not a lot) see information here or restore from backup (If you have a lot of orphan nodes)

How to find all Orphan Node in CRX

You often come across following error in CRX
NodeState CHILD_UUID references inexistent parent uuid PARENT_UUID
Unfortunately you can not fix this issue. But at least you can see that they exist in your repo.

Drop the attached JSP in runtime/0/_crx/diagnostic/, start CRX/CQ and try it to run through http://:/crx/diagnostic/showNodes.jsp. You can specify a path to a tar file or copy folder. It will scan through all nodes with uuid (referenceable)
and output an exception if the node is orphaned.

Other Option If you are sure that you have all the orphan node log message in crx log file, You can use any tool to parse them and find.

Thanks Andrew Khoury and Henry Seginor from Adobe for providing this information

Which index is what in CQ/CRX

> /repository/repository/index
Lucene Search Index for jcr:system (includes jcr:versionStorage)

> /repository/workspace/crx.default/index
Lucene Search Index for crx.default workspace

> /repository/workspace/crx.default/index*.tar
Tar PM Index for crx.default workspace

> /version/index {Version index offcourse}
Tar PM Index for version workspace

> /tarJournal/index*.tar
Tar PM Index for the cluster journal for Journal PM in 5.4

Thanks Andrew Khoury From Adobe for providing this information

How to find and remove all .lock file in CQ

Why: Sometime even after stopping CQ instance all the lock file does not get deleted, Causing problem in restart.

Use following command

find <path to /crx-quickstart> -name "\.lock" -exec rm '{}' \; -print

Simple unix stuff huh :)

How to fix LazyTextExtractorField: Failed to extract text from a binary property (LazyTextExtractorField.java, line 180) error

Symptom : Upload of certain documents (word, excel, pdf) cause system to go down because of OOM error. And since these document are under workflow, Restart of CQ does not help either.

Resolution :

1) Add -Dcom.day.crx.persistence.tar.IndexMergeDelay=0 In start up script. This will make sure that after uploading any document or changing large properties synchronize merge (There is known issue with Synchronize Index Merging) should not cause any issue.

2) Modify your repository.xml and in workspace.xml

change

<SearchIndex class="com.day.crx.query.lucene.LuceneHandler">
<param name="path" value="${wsp.home}/index"/>
<param name="resultFetchSize" value="50"/>
</SearchIndex>

to

<SearchIndex class="com.day.crx.query.lucene.LuceneHandler">
<param name="path" value="${wsp.home}/index"/>
<param name="resultFetchSize" value="50"/>
<param name="forkJavaCommand" value="nice java -Xmx32m"/>
<param name="extractorPoolSize" value="2"/>
</SearchIndex>


For above option in CRX2.1, you have to make sure that Hotfix pack 2.1.0.9 is installed. It will work OOTB in CRX2.2.

Please note that this will not help those file to process successfully, but help you to get rid of OOM error and your system will not go down.

Other Option: If you are really not concern about full text indexing of these documents, You could disable indexing of these document in tika-config (crx-quickstart/server/runtime/0/_crx/WEB-INF/classes/org/apache/jackrabbit/core/query/lucene/tika-config.xml). If this folder structure is not present then you have to create one. Original tika_config.xml can be found by unzipping crx-quickstart/server/runtime/0/_crx/WEB-INF/libs/jackrabbit-core-*.jar (Copy it to some other location, rename it to .zip and then unzip) and then going to org/apache/jackrabbit/core/query/lucene).

You could add org.apache.tika.parser.EmptyParser as class for not to parse document type.

For example (To not index excel sheet)

<parser class="org.apache.tika.parser.EmptyParser">
<mime>application/vnd.openxmlformats-officedocument.spreadsheetml.sheet</mime>
</parser>

To remove PDF parsing you can remove entry
<parser name="parse-pdf" class="org.apache.tika.parser.pdf.PDFParser">
<mime>application/pdf</mime>
</parser>

Above method will also help you to reduce Index size (Lucene) in CQ.

Note: To reduce Lucene Index size you can also add following in workspace.xml

<SearchIndex class="com.day.crx.query.lucene.LuceneHandler">
<param name="path" value="${wsp.home}/index"/>
<!-- add below param -->
<param name="supportHighlighting" value="false"/>
</SearchIndex>