How to change all password using curl in CQ / WEM (Till CQ5.4)

=====================================
#!/bin/bash

OLD_PWD=admin
NEW_PWD=admin
CONSOLE_PWD=admin
HOST=localhost
PORT=4502
echo changing crx admin password:
curl --data rep:password=$NEW_PWD --user admin:admin http://$HOST:$PORT/home/users/a/admin

echo change cqse admin password:
curl --data username="admin" --data password_old=$OLD_PWD --data password=$NEW_PWD --data password_check=$NEW_PWD --user admin:admin http://$HOST:$PORT/admin/passwd

echo change replication agent password:
curl --data transportPassword=$NEW_PWD --user admin:$OLD_PWD http://$HOST:$PORT/etc/replication/agents.author/publish/jcr:content

echo change sling crx client password:
echo below script does:
echo 1. find specific pid (needs https://github.com/micha/jsawk)
echo 2. set password property for that pid
FACTORY_PID=com.day.crx.sling.client.impl.CRXSlingClientRepository

PID=`curl -s -u admin:$CONSOLE_PWD "http://$HOST:$PORT/system/console/configMgr/(service.factoryPid=$FACTORY_PID).json" | jsawk -n 'out(this.pid)'`
echo ">> $PID"
curl -s -u admin:$CONSOLE_PWD -dapply=true -dadmin.password=$NEW_PWD -dpropertylist=admin.password http://$HOST/system/console/configMgr/$PID > /dev/null

=====================================


Note: Have not tested it. Please test it before use.

Thanks Alexander Klimetschek from Adobe for this information.

How to truncate and interpret Tar File in CQ / WEM

Use Case: You want to analyze what is there in tar file or possibly truncate it using tar file truncate tools.

Solution:

Soppose you try to analyzing very first data tar file under (/crx-quickstart/repository/workspace/crx.default)

$ tar -tvf data_00010.tar | less

-rwxrwxrwx 0/0 301 2011-11-22 02:00 cafebabe-cafe-babe-cafe-babecafebabe.na
-rwxrwxrwx 0/0 80 2011-11-22 02:00 deadbeef-cafe-babe-cafe-babecafebabe.na
-rwxrwxrwx 0/0 142 2011-11-22 02:00 f34600a5-be47-41be-8568-8c79dcbd7355.na
-rwxrwxrwx 0/0 296 2011-11-22 02:00 21232f29-7a57-35a7-8389-4a0e4a801fc3.na
-rwxrwxrwx 0/0 348 2011-11-22 02:00 a9c3313a-c296-486d-99ac-b56ebfdefd26.na
-rwxrwxrwx 0/0 192 2011-11-22 02:00 63781579-2ec1-441d-b582-3eea7ce6a0bd.na
-rwxrwxrwx 0/0 276 2011-11-22 02:00 294de355-7d9d-30b3-92d8-a1e6aab028cf.na
-rwxrwxrwx 0/0 286 2011-11-22 02:00 984c30b3-96f5-48b7-a1f3-3d74d0f00460.na
-rwxrwxrwx 0/0 0 2011-11-23 11:30 1322065821400/2782dd9a-c5a8-4689-a9aa-ae46a4074ac3.n
.....
-rwxrwxrwx 0/0 20 2011-11-22 02:00 commit-1321945201521.sh

Entry without <number>/<UUID> is system UUID (First one being root node) and should not be modified.

The entries with 1322065821400/2782dd9a-c5a8-4689-a9aa-ae46a4074ac3.n mean the transaction was started at Wed Nov 23 2011 11:30:21 GMT-0500 (EST) (= System.out.println(new java.sql.Timestamp(1299071667963L).toString()) or http://www.esqsoft.com/javascript_examples/date-to-epoch.htm) on UUID 2782dd9a-c5a8-4689-a9aa-ae46a4074ac3 then

entry with commit-1321945201521.sh mean transaction committed at Tue Nov 22 2011 02:00:01 GMT-0500 (EST)

To truncate a tar file you need Hex Editor in combination with dd tool.

Something like hexdump -C data_00010.tar | more -- To get Hex Location of time stamp

and then dd if=./data_00010.tar of=./data_00010_copy.tar bs=<location> count=1 -- To truncate. Note that "Location" is decimal value so you might need to convert Hex to Decimal to use this command.


For example
$ tar -tvf data_00010.tar | tail -4f
-rwxrwxrwx 0/0 20 2011-11-23 17:24 commit-1322087067980.sh
-rwxrwxrwx 0/0 0 2011-11-23 18:20 1322090435044/398a250e-96f7-4029-a083-4c78f426e56d.n
-rwxrwxrwx 0/0 203 2011-11-23 18:20 1322090435044/7efe2484-5818-4638-8ccb-08ad9c2fe4c0.na
-rwxrwxrwx 0/0 20 2011-11-23 18:20 commit-1322090435044.sh

$ hexdump -C data_00010.tar | tail -100f

03ac3400 63 6f 6d 6d 69 74 2d 31 33 32 32 30 38 37 30 36 |commit-132208706|
03ac3410 37 39 38 30 2e 73 68 00 00 00 00 00 00 00 00 00 |7980.sh.........|
03ac3420 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
03ac3460 00 00 00 00 20 20 20 37 37 37 20 00 20 20 20 20 |.... 777 . |
03ac3470 20 30 20 00 20 20 20 20 20 30 20 00 20 20 20 20 | 0 . 0 . |
03ac3480 20 20 20 20 20 32 34 20 31 31 36 36 33 32 37 31 | 24 11663271|
03ac3490 32 33 33 20 20 20 20 37 32 32 32 20 30 00 00 00 |233 7222 0...|
03ac34a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
03ac3600 6d 76 20 31 33 32 32 30 38 37 30 36 37 39 38 30 |mv 1322087067980|
03ac3610 2f 2a 20 2e 00 00 00 00 00 00 00 00 00 00 00 00 |/* .............|
03ac3620 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
03ac3800 31 33 32 32 30 39 30 34 33 35 30 34 34 2f 33 39 |1322090435044/39|
03ac3810 38 61 32 35 30 65 2d 39 36 66 37 2d 34 30 32 39 |8a250e-96f7-4029|
03ac3820 2d 61 30 38 33 2d 34 63 37 38 66 34 32 36 65 35 |-a083-4c78f426e5|
03ac3830 36 64 2e 6e 00 00 00 00 00 00 00 00 00 00 00 00 |6d.n............|
03ac3840 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
03ac3860 00 00 00 00 20 20 20 37 37 37 20 00 20 20 20 20 |.... 777 . |
03ac3870 20 30 20 00 20 20 20 20 20 30 20 00 20 20 20 20 | 0 . 0 . |
03ac3880 20 20 20 20 20 20 30 20 31 31 36 36 33 32 37 37 | 0 11663277|
03ac3890 37 30 33 20 20 20 31 32 31 31 30 20 30 00 00 00 |703 12110 0...|
03ac38a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

Now in above example suppose you want to truncate on 1322090435044/398a250e-96f7-4029-a083-4c78f426e56d.n Now see corresponding entry in Hex output

03ac3800 31 33 32 32 30 39 30 34 33 35 30 34 34 2f 33 39 |1322090435044/39|
03ac3810 38 61 32 35 30 65 2d 39 36 66 37 2d 34 30 32 39 |8a250e-96f7-4029|
03ac3820 2d 61 30 38 33 2d 34 63 37 38 66 34 32 36 65 35 |-a083-4c78f426e5|
03ac3830 36 64 2e 6e 00 00 00 00 00 00 00 00 00 00 00 00 |6d.n............|
03ac3840 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

Then get first value of hex that is "03ac3800" in this case and convert it to decimal using any tool (http://www.binaryhexconverter.com/hex-to-decimal-converter). Decimal value would be "61618176". Then use following command to truncate

dd if=./data_00010.tar of=./data_00010_copy.tar bs=61618176 count=1

Now if you will examine both file (See data tar file is truncated (data_00010_copy.tar) and original file is under data_00010.tar)

$ tar -tvf data_00010.tar | tail -6f
-rwxrwxrwx 0/0 20 2011-11-23 16:09 commit-1322082572919.sh
-rwxrwxrwx 0/0 168 2011-11-23 17:24 1322087067980/fb042de1-85f4-4ca3-b4ee-e083ea79753c.na
-rwxrwxrwx 0/0 20 2011-11-23 17:24 commit-1322087067980.sh
-rwxrwxrwx 0/0 0 2011-11-23 18:20 1322090435044/398a250e-96f7-4029-a083-4c78f426e56d.n
-rwxrwxrwx 0/0 203 2011-11-23 18:20 1322090435044/7efe2484-5818-4638-8ccb-08ad9c2fe4c0.na
-rwxrwxrwx 0/0 20 2011-11-23 18:20 commit-1322090435044.sh
$ tar -tvf data_00010_copy.tar | tail -6f
-rwxrwxrwx 0/0 20 2011-11-23 12:14 commit-1322068449556.sh
-rwxrwxrwx 0/0 0 2011-11-23 16:09 1322082572919/ef65dcfa-000a-4ede-ab3d-4844ee66d242.n
-rwxrwxrwx 0/0 92 2011-11-23 16:09 1322082572919/97daadf7-770b-4005-a1f1-86040b32b9b5.na
-rwxrwxrwx 0/0 20 2011-11-23 16:09 commit-1322082572919.sh
-rwxrwxrwx 0/0 168 2011-11-23 17:24 1322087067980/fb042de1-85f4-4ca3-b4ee-e083ea79753c.na
-rwxrwxrwx 0/0 20 2011-11-23 17:24 commit-1322087067980.sh


1) http://www.binaryhexconverter.com/hex-to-decimal-converter
2) http://www.novell.com/communities/node/6419/making-sense-hexdump
3) http://en.wikipedia.org/wiki/Dd_(Unix)

Thanks Thomas Mueller From Adobe for providing this information.

Caution: This command is very dangerous and lead to data loss. Please use it with "A lot" of caution.

How to write CQ / WEM init script for Linux

Use case : You want to start and stop CQ on system start and stop

Solution: Here is script that you can use. Put this script in /etc/init.d


1. Create a file /etc/init.d/cq5 on your server and copy the contents from below script.
2. edit the file and make the following changes
update the "pidfile:" config to match your cq instance
update the "CQ5_ROOT" and "CQ5_USER" variables to match your cq instance
3. Run this command:
chmod 755 /etc/init.d/cq5
4. Run this command to add the config to redhat startup and shutdown:
chkconfig --add cq5

#!/bin/bash
#
# /etc/rc.d/init.d/cq5
#
#
#
#
# # of the file to the end of the tags section must begin with a #
# character. After the tags section, there should be a blank line.
# This keeps normal comments in the rest of the file from being
# mistaken for tags, should they happen to fit the pattern.>
#
# chkconfig: 35 85 15
# description: This service manages the Adobe WEM Content Management java process.
# processname: cq5
# pidfile: /crx-quickstart/server/logs/cq.pid

# Source function library.
. /etc/rc.d/init.d/functions

CQ5_ROOT=/opt/cq5
CQ5_USER=cq5

########
SERVER=${CQ5_ROOT}/crx-quickstart/server
START=${SERVER}/start
STOP=${SERVER}/stop
STATUS="${SERVER}/serverctl status"

case "$1" in
start)
echo -n "Starting cq5 services: "
su - ${CQ5_USER} ${START}
touch /var/lock/subsys/cq5
;;
stop)
echo -n "Shutting down cq5 services: "
su - ${CQ5_USER} ${STOP}
rm -f /var/lock/subsys/cq5
;;
status)
su - ${CQ5_USER} ${STATUS}
;;
restart)
su - ${CQ5_USER} ${STOP}
su - ${CQ5_USER} ${START}
;;
reload)
;;
*)
echo "Usage: cq5 {start|stop|status|reload}"
exit 1
;;
esac




For CQ5.5



#!/bin/bash
#
# /etc/r.d/init.d/cq5author
#
# of the file to the end of the tags section must begin with a #
# character. After the tags section, there should be a blank line.
# This keeps normal comments in the rest of the file from being
# mistaken for tags, should they happen to fit the pattern.>
#
# chkconfig: 35 85 15
# description: This service manages the Adobe WEM Content Management java process.
# processname: cq5author
# pidfile: /crx-quickstart/conf/cq.pid

# Source function library.
. /etc/rc.d/init.d/functions

CQ5_ROOT=/opt/Adobe/cq5_5/author
CQ5_USER=root

########
SERVER=${CQ5_ROOT}/crx-quickstart
START=${SERVER}/bin/start
STOP=${SERVER}/bin/stop
STATUS="${SERVER}/bin/status"

case "$1" in
start)
echo -n "Starting cq5 services: "
su - ${CQ5_USER} ${START}
touch /var/lock/subsys/cq5
;;
stop)
echo -n "Shutting down cq5 services: "
su - ${CQ5_USER} ${STOP}
rm -f /var/lock/subsys/cq5
;;
status)
su - ${CQ5_USER} ${STATUS}
;;
restart)
su - ${CQ5_USER} ${STOP}
su - ${CQ5_USER} ${START}
;;
reload)
;;
*)
echo "Usage: cq5 {start|stop|status|reload}"
exit 1
;;
esac

Thanks Andrew Khoury and Daniel Gordon from Adobe for providing this information.

How to mount DataStore in different directory in CQ / WEM

Use Case : You have huge datastore and want to mount it on different directory Or Online backup takes a lot of time including datastore and you want to do separate backup for datastore

Solution : Here is process for how to separate Datastore and then do separate online backup

Assuming your repository is under /mnt/crx/repository and you want to move your datastore to /mnt/crx/datastore

1. stop the crx instance
2. mv /mnt/crx/repository/shared/repository/datastore /mnt/crx/
3. Then modify repository.xml by adding the new path configuration to the DataStore element.

Before:
<DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
<param name="minRecordLength" value="4096"/>
</DataStore>

After:
<DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
<param name="path" value="/mnt/crx/datastore"/>
<param name="minRecordLength" value="4096"/>
</DataStore>

4. Start CRX

After doing this then you can safely run separate backups on the datastore while the system is running without affecting performance very much.

Following our example, you could use this command to backup the datstore:
rsync --av --ignore-existing /mnt/crx/datastore /mnt/backup/datastore

Information about rsync can be found here http://ss64.com/bash/rsync.html

How to create package based on Xpath in CQ5 / WEM

Use Case : Some time you want to create package based on Xpath. CQ5 package manager does not have ability to create package based on Xpath.

Solution :

You can use following package to achieve this

FOR CQ5.3 and CQ5.4



FOR CQ5.5



FOR CQ5.6 (Also fixes some other issues)



FOR CQ6

Above package might not work for AEM 6 onward because of this feature You might have to do disable POST from felix console for CSRF token. Add it back after running this tool.

1) Download and Install package using package manager
2) go to <host>:<port>/apps/tools/components/createPackage/run.html
3) Give your Xpath in xpath value
4) You can also add comma separate exclude path that you don't want to add to package.
5) Click on Create config package
6) Now Download the package and also be saved under /etc/packages/CQSupportTool

For example if you have to create package of all ACL to migrate from one CQ instance to another you can use xpath query for package as //element(*,rep:ACL)

Please note that, this package is for test purpose only. Feel free to modify it based on your need.

Known Issue Exception when "/" root is given (I will fix that as soon as get some time).

How to fix com.day.crx.cluster.LoginException: Slave has same identity as master Error

Symptom: You get Caused by: com.day.crx.cluster.LoginException: Slave has same identity as master error and CQ does not start up. This usually happen in CQ5.3 or upgraded CQ5.4 instance.

Solution :

-Stop instance
-Delete all files named listener.properties in

* crx-quickstart/repository/shared/journal/control
* crx-quickstart/repository/shared/version/control
* crx-quickstart/repository/shared/workspaces//control

- Delete file crx-quickstart/repository/cluster_node.id
- Delete all stale .lock files (if existing)
- Delete all stale lock.properties files (if existing)
- Start CQ5.3 instance

OR
1) Use following following command to remove all listener.properties files
find ./crx-quickstart/repository/shared -name listener.properties | xargs rm -f

2) - Delete file crx-quickstart/repository/cluster_node.id

3) Delete all .lock files (if existing) using
find ./crx-quickstart/repository -name "\.lock" -exec rm '{}' \; -print

How to fix File Not found issue in CRX

Caution : Most of time it works but not guaranteed

Assumption : You are using CRX and latest HF is not installed. Otherwise need to figure out why you had this issue.

Symptom : You get following error in log

Caused by: org.apache.jackrabbit.core.state.ItemStateException: Failed to read bundle: <SOME -- UUID> : java.io.IOException: File not found: 263
at com.day.crx.persistence.tar.TarPersistenceManager.getInputStream(TarPersistenceManager.java:1195)
at com.day.crx.persistence.tar.TarPersistenceManager.loadBundle(TarPersistenceManager.java:334)
at com.day.crx.persistence.tar.TarPersistenceManager.loadBundle(TarPersistenceManager.java:311)
at org.apache.jackrabbit.core.persistence.bundle.AbstractBundlePersistenceManager.getBundle(AbstractBundlePersistenceManager.java:654)
at org.apache.jackrabbit.core.persistence.bundle.AbstractBundlePersistenceManager.load(AbstractBundlePersistenceManager.java:400)
at org.apache.jackrabbit.core.state.SharedItemStateManager.loadItemState(SharedItemStateManager.java:1819)
at org.apache.jackrabbit.core.state.SharedItemStateManager.getNonVirtualItemState(SharedItemStateManager.java:1739)
at org.apache.jackrabbit.core.state.SharedItemStateManager.getItemState(SharedItemStateManager.java:261)
at org.apache.jackrabbit.core.state.LocalItemStateManager.getNodeState(LocalItemStateManager.java:107)
at org.apache.jackrabbit.core.state.LocalItemStateManager.getItemState(LocalItemStateManager.java:172)
at org.apache.jackrabbit.core.state.XAItemStateManager.getItemState(XAItemStateManager.java:260)
at org.apache.jackrabbit.core.state.SessionItemStateManager.getItemState(SessionItemStateManager.java:161)
at org.apache.jackrabbit.core.ItemManager.getItemData(ItemManager.java:370)
... 34 more
Caused by: java.io.IOException: File not found: 263
at com.day.crx.persistence.tar.TarSet.getInputStream(TarSet.java:731)
at com.day.crx.persistence.tar.TarSet.getInputStream(TarSet.java:724)
at com.day.crx.persistence.tar.ClusterTarSet.getInputStream(ClusterTarSet.java:502)
at com.day.crx.persistence.tar.TarPersistenceManager.getInputStream(TarPersistenceManager.java:1191)
... 46 more



Solution

1) stop the instance
2) Make sure that it stopped (You can run ps -ef | grep java command to check that)
3) delete [take backup] all index_*.tar files found in /crx-quickstart/repository/workspaces/crx.default/copy (If it is CQ5.3 or upgraded instance)
4) Grep the error.log to find the missing tar file numbers:
grep "^java.io.IOException: File not found:" error.log* | awk '{ print $5 }' | sort –u
5) from your nightly backups Recover and place missing tar file "Data*.tar" at repository/workspace/crx.default/
6) Set the file permissions back to rw as the system sets all data tar files to read only after a file is missing. chmod 755 *.tar.
7) Start CRX / CQ with the following system property (in start up script):
java -Dcom.day.crx.persistence.tar.IndexMergeDelay=0
8) Make sure that you have latest CRX Hotfix installed


If above issue happen in clustered environment, You can recover it from working cluster instance. See How to recover from Out of sync node in cluster

Please do not attempt to do this directly on production instance without taking help of Adobe Support

How to remove lock files in CQ / WEM

Problem :

javax.jcr.RepositoryException: The repository home /crx-quickstart/repository appears to be in use since the file named .lock is locked by another process.
at org.apache.jackrabbit.core.util.RepositoryLock.tryLock(RepositoryLock.java:166)
at org.apache.jackrabbit.core.util.RepositoryLock.acquire(RepositoryLock.java:138)
at org.apache.jackrabbit.core.RepositoryImpl.(RepositoryImpl.java:276)
at com.day.crx.core.CRXRepositoryImpl.(CRXRepositoryImpl.java:233)
at com.day.crx.core.CRXRepositoryImpl.(CRXRepositoryImpl.java:288)
at com.day.crx.core.CRXRepositoryImpl.create(CRXRepositoryImpl.java:193)
at com.day.crx.j2ee.CRXRepositoryStartupServlet.createRepository(CRXRepositoryStartupServlet.java:173)

Reason why above happen, If you forcefully shut down CQ sometimes .lock files does not get deleted.

Solution :

You can use following command in Linux/ Unix platform to delete all lock files

find <Path to crx-quickstart>/repository -name "\.lock" -exec rm '{}' \; -print

How to use SVN with VLT in CQ / WEM

Use Case : You can not do all the operation using svn with CRXDE or CRXDE light. Also if you are using some other development tool, You can use File system to perform SVN operations.

Pre requisite : You have svn and vlt already set up
For svn set up use svn product guideline
for vlt set up refer http://dev.day.com/docs/en/crx/current/how_to/how_to_use_the_vlttool.html

Solution :

Initial Set up :
$ cd /projects
$ svn mkdir http://svn.server.com/repos/myproject
$ svn co http://svn.server.com/repos/myproject
$ vlt export -v http://localhost:4502/crx /apps/geometrixx geometrixx
$ cd geometrixx/
$ svn add META-INF/ jcr_root/
$ svn ci

Changes made in File System and update to local repository (JCR)  If you are using vlt sync then you don't have to do this:

$ cd /projects/geometrixx
$ vlt -v import http://localhost:4502/crx . /
Or
$ vlt add <File you added>
$ vlt ci
Or
$ vlt st
$ vlt ci

Changes made in SVN and update to File system :
$ cd /projects/geometrixx
$ svn st
M META-INF/vault/properties.xml
M jcr_root/apps/geometrixx/components/contentpage/.content.xml
$ svn up
If you are using vlt sync then you don't have to use this option
$ vlt ci


Changes made in repository and update it back to file system (With vlt sync you don't have to use this)
1) go to: content/jcr_root/<Path>
2) vlt up --force


Changes made in a file in local file system and update repository:

For example you are changing one file

1) go to: .../jcr_content/libs/foundation/components/text
2) edit something in text.jsp
3) see the modified files vlt st
4) see the changes: vlt diff text.jsp
5) commit the changes: vlt ci test.jsp
6) reload page containing a text component and see that it worked



So direction could be any thing (After initial set up, You don't have to do it when using vlt sync)

repo change ---> vlt st ---> vlt up --> svn st --> svn ci

svn change ----> svn st --> svn up --> vlt ci

file system change ---> vlt st ---> vlt ci And then svn st ----> svn ci

Now with vlt synch option you can synch between your eclipse project and crx. This helps not to use vlt ci every time you make file system changes and use vlt up when you make crxde changes.

More infirmation about vlt synch can be found here http://dev.day.com/docs/en/crx/current/how_to/how_to_use_the_vlttool.html#Using%20VLT%20sync%20for%20development