How to Fix No Such Item repository corruption in CQ5.5

Use Case: You have repository corruption that can not be fixed using consistency check and fix.

Prerequisite:

http://helpx.adobe.com/crx/kb/RepositoryInconsistency.html

http://helpx.adobe.com/cq/kb/cq-5-5--rien-ne-va-plus---i-need-a-text-shell.html

Stack Trace:

05.11.2012 00:30:01.476 *ERROR* [pool-5-thread-2] Unable to perform role import javax.jcr.ItemNotFoundException: failed to build path of 795cff24-7fd7-3f5f-a9b1-4724c3a6ff4d
        at org.apache.jackrabbit.core.HierarchyManagerImpl.getPath(HierarchyManagerImpl.java:400)
        at org.apache.jackrabbit.core.CachingHierarchyManager.getPath(CachingHierarchyManager.java:233)
        at org.apache.jackrabbit.core.ItemImpl.getPrimaryPath(ItemImpl.java:188)
        at org.apache.jackrabbit.core.NodeImpl.getPrimaryPath(NodeImpl.java:2753)
        at org.apache.jackrabbit.core.ItemImpl$2.perform(ItemImpl.java:379)
        at org.apache.jackrabbit.core.ItemImpl$2.perform(ItemImpl.java:376)
        at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:216)
        at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91)
        at org.apache.jackrabbit.core.ItemImpl.getPath(ItemImpl.java:376)
        at <Some Custom Code>
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.jackrabbit.core.state.NoSuchItemStateException: 34228aff-f648-429a-a097-29de8568fe3e
        at org.apache.jackrabbit.core.state.SharedItemStateManager.getItemState(SharedItemStateManager.java:285)
        at org.apache.jackrabbit.core.state.LocalItemStateManager.getNodeState(LocalItemStateManager.java:110)
        at org.apache.jackrabbit.core.state.LocalItemStateManager.getItemState(LocalItemStateManager.java:175)
        at org.apache.jackrabbit.core.state.XAItemStateManager.getItemState(XAItemStateManager.java:260)
        at org.apache.jackrabbit.core.state.SessionItemStateManager.getItemState(SessionItemStateManager.java:161)
        at org.apache.jackrabbit.core.HierarchyManagerImpl.getItemState(HierarchyManagerImpl.java:152)
        at org.apache.jackrabbit.core.HierarchyManagerImpl.buildPath(HierarchyManagerImpl.java:278)
        at org.apache.jackrabbit.core.CachingHierarchyManager.buildPath(CachingHierarchyManager.java:199)
        at org.apache.jackrabbit.core.HierarchyManagerImpl.getPath(HierarchyManagerImpl.java:395)
        ... 18 more

How to Fix (If it is not cluster then ignore cluster steps):

1) Stop all slave
2) In master create a folder called install under crx-quickstart
3) Copy files for gogo shell in install folder (From Here)
4) Use command telnet localhost 6666 from instance or from felix console go to gogo tab

5) Use command g! "login admin <admin password>"
6) Use command "rm [<UUID of Node you are having Problem in above stack trace it is 795cff24-7fd7-3f5f-a9b1-4724c3a6ff4d>]" then use command "save"
7) Then go to crx-quickstart/repository/workspace/crx.default
8) Create a file called redo_X.log where X is same as latest index_X.tar. for example if latest index tar file name is index_9_24.tar then create a file called redo_9_24.log
9) Add following entry in redo log (Based on above stack trace, This is to recreate search Index)
10) 1 DEL 795cff24-7fd7-3f5f-a9b1-4724c3a6ff4d
      1 DEL 34228aff-f648-429a-a097-29de8568fe3e
      1 COM
11) Restart master instance
12) Start slave instance

Note: Before you delete any node, You can do stat on that node to see what node is causing problem. You can do something like,

g! stat 795cff24-7fd7-3f5f-a9b1-4724c3a6ff4d
Path:         {} {}34228aff-f648-429a-a097-29de8568fe3e
NodeId:       795cff24-7fd7-3f5f-a9b1-4724c3a6ff4d    
ParentId:     34228aff-f648-429a-a097-29de8568fe3e    
NodeTypeName: {internal}Group                        
Memory used:  2400                                    
ChildNode Entries:
+ {internal}policy[1] -> c577627a-bb22-4c82-95af-6acc1881302a

Above output shows node is pointing to orphan node, Which can not be fixed by consistency check and fix.

Use help command to find things that you can do with gogo shell.

Caution: gogo shell (Similar to crx console in earlier version) is very powerful tool and cause system damage as well. Please use this tool with care.

Let me know if you want detail about any command in gogo shell.