How to fix com.day.crx.persistence.tar.ClusterTarSet Could not open java.io.IOException: Bad file descriptor Issue in CQ

Issue: Server does not start and you see following error in logs

*WARN* [FelixStartLevel] com.day.crx.persistence.tar.ClusterTarSet Could not open java.io.IOException: Bad file descriptor

at java.io.RandomAccessFile.writeBytes(Native Method)

at java.io.RandomAccessFile.write(RandomAccessFile.java:486)

at com.day.crx.persistence.tar.file.TarFile.write(TarFile.java:742)

at com.day.crx.persistence.tar.file.TarFile.writeData(TarFile.java:635)

at com.day.crx.persistence.tar.file.TarFile.appendMetaData(TarFile.java:709)

at com.day.crx.persistence.tar.file.TarFile.append(TarFile.java:591)

at com.day.crx.persistence.tar.TarSet.switchDataFile(TarSet.java:445)

at com.day.crx.persistence.tar.TarSet.open(TarSet.java:227)

at com.day.crx.persistence.tar.ClusterTarSet.reopenCopy(ClusterTarSet.java:1455)

at com.day.crx.persistence.tar.ClusterTarSet.open(ClusterTarSet.java:860)

at com.day.crx.persistence.tar.TarPersistenceManager.openTarSet(TarPersistenceManager.java:980)

at com.day.crx.persistence.tar.TarPersistenceManager.init(TarPersistenceManager.java:500)

at com.day.crx.core.CRXRepositoryImpl.createVersionManager(CRXRepositoryImpl.java:869)

at org.apache.jackrabbit.core.RepositoryImpl.<init>(RepositoryImpl.java:311)

at com.day.crx.core.CRXRepositoryImpl.<init>(CRXRepositoryImpl.java:307)

at com.day.crx.core.CRXRepositoryImpl.create(CRXRepositoryImpl.java:262)

at com.day.crx.core.CRXRepositoryImpl.create(CRXRepositoryImpl.java:245)

at com.day.crx.sling.server.impl.jmx.ManagedRepository.activate(ManagedRepository.java:170)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:236)

at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:37)

at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:613)

at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:496)

at org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:149)

Solution: 

1) Find a line above this error, it would be something like

09.12.2013 13:42:32.030 *INFO* [FelixStartLevel] com.day.crx.persistence.tar.TarSet scanning index <some path>/crx-quickstart/repository/<either version or workspace>/data_<some number>.tar 

2) Based on path go to that location.

3) STOP YOUR INSTANCE. remove all index files using rm -rf <Path from above>/index*tar

4) change permission of data tar file using chmod 644 <path from above>/data*tar

5) Start instance

6) Some cases data tar files can not be recovered. Please check my other post to fix non recovery data tar files.

Caution: If there are a lot of data tar files, Index creation may take some time. Please create daycare ticket to find root cause of this issue.