How to build your own CMS [Episode 1 - Justification]

Past few years, I have been working with my team to build our own CMS for marketing content or SEO content, an audacious goal given that we were already using AEM, an enterprise level CMS with great integration with Adobe marketing cloud. In these blog post I will walk through mental model and architecture of how we went about building it.

In building any new product which is not part of your core business and for which good solution already exist, first question you gotta ask yourself or answer to others WHY BUILD ? or WHY NOT JUST BUY ?

I have seen multiple frameworks to help you decide which path to take (build vs buy), in our case since we were already using AEM we already knew some pain point (see below) with framework that help us make decision.  When your company is very engineering driven and have frameworks, platform and system in place that very few has.Our criteria of justifying to leadership came down to 4 things,

Is this problem unique to you: In this case NO. CMS is not a unique problem, many company has CMS needs for different use case and there is plethora of solution that exist for those use cases. 

Is this part of your core business: YES and NO, Core CMS feature it self is not part of Core business, Company is not going to build and sell enterprise level CMS. However there are certain product which uses subset of CMS features (we will get to that later) 

Does this need white glove support: If you buy you definitely need white glove support, specially with AEM getting support is harder, hiring someone who knows AEM or really good at it is even harder. And even if you hire them it hard to have a career path for them in other part of company. We struggled with hiring and retaining AEM talent.

Integration: This was BIG one, if most of your app are custom integrating with third party solution is harder, all the way from using Data (Internal data), Infrastructure (using physical or virtual hardware running in same network), CI/CD (all the way from code hosting to canary testing to prod deployment), Accessibility, Security, Designs (align with visual design of rest of company and keeping system up to date) etc. We continuously struggled with integrating AEM with company stack, processes and infrastructure. 

After using AEM for years we stack rank these criteria against our future need and decided to build our own CMS.

Well so far so good, but how would you go about justifying this to leadership and most important executing it. I have seen many companies trying it and not succeeding so it was big risk. In section two I will try to cover steps we took to go from justification to POC [Episode 2 - Proof of Concept] 

AEM pain points:

• Not a true Microservice: Given that how author, publish and communication between two (through replication) works, I don't think it is true microservice. It is really hard to scale AEM horizontally (both author and publish)
• Technology & Ramp up: Now this could be debatable but I think AEM tech stack is not widely used teach stack, for example app request processing (sling), Sightly (Templating language), Event handling (OSGI or sling events), Data replication (Proprietary), persistence (Tar), Apart from backend being in Java none of the other technologies are widely used and hence hard to find generalist and hard to ramp someone on this product. 
• High reliant on dispatcher: Effective caching is still not a solved problem and AEM rely a lot on effective caching at dispatcher to do app level scaling. We could not scale system if you are service dynamic data.
• Automation: AEM deployment pipeline is tightly coupled, there are only few options you have to extend it, it makes it hard to integrate it with existing systems. 
• Code management: Fact that actual code (front end) and bundles lives where content also lives do not provide good isolations and hard to do code management (versioning etc) and plug in in to existing deployment pipeline.
• Upgrade: Not sure how many of you faced this but upgrading AEM systems are not easy and takes a lot of resources. Engineers end of spending more time on upgrade than developing features.

  

How to start sling with RDBMS

Use case: Start Sling using RDBMS (Mysql, Oracle) as a persistent manager.

Prerequisite: Make sure that you have mysql or oracle install. Here is instruction https://dev.mysql.com/doc/mysql-installation-excerpt/5.7/en/

Currently sling comes out of the box with Tar and Mongo persistent manager but setting up RDBMS instruction is hard to find.

Note that sling underneath has Oak for data persistence and OAK does support different type of RDB persistence that you can get it from here. https://jackrabbit.apache.org/oak/docs/nodestore/document/rdb-document-store.html

Here are steps to set up RDB persistence for sling,


Step 1: Checkout sling project from https://sling.apache.org/downloads.cgi
example

git clone https://github.com/apache/sling-org-apache-sling-starter.git

OR

git clone https://github.com/apache/sling-org-apache-sling-app-cms.git

Step 2: find oak.txt either under src/main/provisioning or builder/src/main/provisioning

Step 3: Add the following lines in oak.txt

[artifacts startLevel=15 runModes=oak_rdb]
mysql/mysql-connector-java/8.0.20
com.h2database/h2-mvstore/1.4.194
org.apache.sling/org.apache.sling.datasource/1.0.4

org.apache.sling.datasource.DataSourceFactory-mysql.config 
url="jdbc:mysql://localhost:3306/slingcms"         
driverClassName="com.mysql.jdbc.Driver"   
username="root"   
password=""   
datasource.name="oak"
validationQuery="show\ tables"
connectionProperties="serverTimezone\=UTC"

Step 4: Find  boot.txt which is under the same folder where oak.txt was and append the following in property sling.run.mode.install.options

sling.run.mode.install.options=oak_tar,oak_mongo,oak_rdb

Step 5: Build project using

mvn clean install

Step 6: Copy stand-alone jar files to the location where you want to run the project (Usually they are under target folder at root or builder directory)

Step 7: Run following command 

java -jar -Dsling.run.modes=oak_rdb <jar file> start

Step 8: To check if right tables are created you can down load any mysql explorer https://www.mysql.com/products/workbench/ and check

How to work with Report in CQ / WEM

Use Case: You want to create custom report for any thing in CQ

Prerequisite: Please read very carefully

• http://dev.day.com/docs/en/cq/current/administering/reporting.html
• http://dev.day.com/docs/en/cq/current/developing/developing-reports.html

 What we are covering:

I will give one custom example and guide you how to create report within CQ for your custom requirement.

Q: Where I can find all CQ OOTB reports.
A: You can find them under /etc/reports. You can also navigate to CQ report using siteadmin -> tools-> /reports -> double click on any report

Q: Where is code for OOTB reporting templates and component
A: They can be found under /libs/cq/reporting and API http://dev.day.com/docs/en/cq/current/javadoc/com/day/cq/reporting/package-summary.html (I don't think you might have to use these API, Unless there is any specific requirement )

If you read above document very carefully, You will find that not all the reports comes with CQ uses CQ reporting framework. Specially Request log parser report and Disk Usage report.

We will cover CQ reporting example here using CQ reporting framework. For any other report please use example of disk usage or log parser report.

Requirement:

Create CQ report to get Geometrixx product page information. Report should include,

1) Title
2) Tags
3) URL
4) Overview Link
5) Feature Link

You should be able to group report based on Tags. You should also be able to see chart representation of report. You should also be able to export report outside CQ instance.


Solution:

First, Make sure that this can be done using CQ reporting framework. For that go through each field for report and make sure that those are available as properties within that page through "Fix" path and there is no complex logic involve to get that data.

Once you know that this can be done using CQ reports, Start developing report.

Step 1: Create template

• Create a template for your report. See example under /libs/cq/reporting/templates. 
• Create node called /apps/cq/reporting/templates
• Easy is to copy existing report from /libs (I copied user report) to /apps and rename it. In our case I created a report called /apps/cq/reporting/templates/productReport
• Go to /apps/cq/reporting/templates/productReport/jcr:content/report and assign resource type for your custom report. Also create a property called "rootPath" and assign value as "/content/geometrixx/en/products"
• Template will look like this.

<?xml version="1.0" encoding="UTF-8"?>

<jcr:root xmlns:sling="http://sling.apache.org/jcr/sling/1.0" xmlns:cq="http://www.day.com/jcr/cq/1.0" xmlns:jcr="http://www.jcp.org/jcr/1.0" xmlns:nt="http://www.jcp.org/jcr/nt/1.0"

    jcr:description="Geometrixx Product Report"

    jcr:primaryType="cq:Template"

    jcr:title="Geometrixx Product Report"

    allowedPaths="[/etc/reports(/.*)?]"

    ranking="{Long}100"

    shortTitle="Product Report">

    <jcr:content

        jcr:primaryType="cq:PageContent"

        sling:resourceType="cq/reporting/components/reportpage">

        <report

            jcr:primaryType="nt:unstructured"

            sling:resourceType="cq/reporting/components/productreport/productreport"

            repVersion="{Long}2"

            rootPath="/content/geometrixx/en/products"/>

    </jcr:content>

</jcr:root>

Step 2: Create Component

• You can see example under /libs/cq/reporting/components/<I copied userreport>. Copy directory structure under /cq/reporting/components and rename it as productreport
• Create / Rename (/libs/cq/reporting/components/userreport/userreport to cq/reporting/components/productreport/productreport)
• Make sure that you have following nodes, /apps/cq/reporting/components/productreport/productreport/charting for creating chart data for grouped items. /apps/cq/reporting/components/productreport/productreport/dialog for existing report dialogs. /libs/cq/reporting/components/userreport/userreport/queryBuilder to specify query that needs to run against the report.
• Change /libs/cq/reporting/components/userreport/userreport/queryBuilder, nodeType property to "cq:PageContent". Because your page jcr:content node has all those properties that we are looking for in our report.
• Please see http://dev.day.com/docs/en/cq/current/developing/developing-reports.html#Query Definition if you want to add additional constraint in query. For example give me only those page whose cq:template value is "Something", In that case you would create something like this

N:queryBuilder
    N:propertyConstraints
    [
        N:N0 // array of nodes (name irrelevant), each with the following properties:
            P:cq:template
            P:something
    ]
    P:nodeTypes cq:PageContent

Step 3: Create all Column

• Once we have query definition to get all pages, Now you can create column to show in your report.
• For that see example under /libs/cq/reporting/components/userreport and see how each column is configured.
• You can copy one column from /libs/cq/reporting/components/userreport under /apps/cq/reporting/components/productreport to start with
• Description about field for a column can be found here http://dev.day.com/docs/en/cq/current/developing/developing-reports.html#Column Base Component
• For example in our case, we want a Tag column. for that create /apps/cq/reporting/components/productreport/productreport/Tagcol with following definition
• Now suppose you want overview page path, for that you have to nevigate to Overview node and then show it. In that case definition would look like this

How to create custom renderer for a file type in AEM

Use Case:

Honestly use cases for this could be different based on what you want to change in response for a specific file type. We will take an example where we want to add some custom header for a PDF file based on what author has added in property of that file. For example here if author has added a canonical url and no index property, then we want to add these property in response header of all pdf file request.

Prerequisite:

Note that in order to make these property available for asset, you have to override asset editor as well. One example of adding extra property to asset editor is here http://www.wemblog.com/2013/01/how-to-associate-cug-with-dam-asset-in.html

Implementation:
Key here is OptingServlet https://sling.apache.org/apidocs/sling7/org/apache/sling/api/servlets/OptingServlet.html which get invoked on every request. Here is one example

Note: Let me know if you have any question about implementation.

How to Write Tests in AEM

Use Case: Writing tests for AEM application.

Current Issue: As your project and code base grows, it is really important to make sure that test coverage for code is there to maintain consistency and sanity of your code. Writing test cases for AEM is little bit different than writing conventional Java test cases, This makes it difficult for beginner to write test cases for AEM application.

Idea of this post to give different options available for writing unit test for AEM services.

Prerequisite:

• Familiar with Java Junit test framework http://junit.org
• Familiar with Java Mockito test framework http://site.mockito.org/
• Optional: PowerMockito test framework http://powermock.github.io/

Good to know:

• Sling test framework (Mock and IT) https://sling.apache.org/documentation/development.html
• AEM Mock http://wcm.io/testing/aem-mock/  and http://wcm.io/testing/aem-mock/apidocs/
• WCMUse classes Mocking http://aempodcast.com/2015/testing/unit-testing-wcmuse-classes-using-mockito/#.WEmiUMMrJKc

I would explain how you can have better test coverage for your application by giving different use cases,

Dependencies: It is recommended to have following dependencies in to your pom before start writing for tests for your application
Case 1: Writing test cases for Generic Helper class.

This is simplest use case where your generic helper class (For example StringUtils, DateUtils) is not using any AEM libraries. For this you can simply use Junit to write your unit test. https://www.tutorialspoint.com/junit

Here is very simple example:


Case 2: Writing test cases for AEM Helper class

This is second use case where you want to test AEM helper methods. For this you can use combination of Junit and Mockito. Use Mockito to Mock AEM services and methods and Junit for assertion.

Here is simple example

Case 3: Writing test cases for AEM services

Now it gets little bit tricky where you need to mock certain behavior of bundle and implicit object. That's why Sling has created Mock version of sling objects and wcm.io has created mock version of AEM objects. You can just use aem mock http://wcm.io/testing/aem-mock/ to achieve most of your use cases. (AEM mock extend Sling mock).

here are some of the common use cases you will come across while testing your service.

1) How can I mock content my service is running against ?

For this it is recommended to use contentLoader API http://wcm.io/testing/aem-mock/usage.html to either load existing json based resource (You can simply get it by creating resource in CRXDE and then using something like RESOURCEPATH.infinity.json to get json for that resource) or just create mock resource using ContentBuilder context.create().resource() or ResourceBuilder context.build().resource() http://wcm.io/testing/aem-mock/apidocs/

Note that if you are mocking Page object then you have to use aem mock using aemcontext.pageManager().create()

2) How Can I initialize properties in the bundle ?

You can use register and activate OSGI service with properties http://wcm.io/testing/aem-mock/usage.html#Registering_OSGi_service for that. Here is some example


3) How Can I inject other service for my service ?

You can either Mock service or use register service API for that http://wcm.io/testing/aem-mock/usage.html#Registering_OSGi_service

Note that when you inject a service to your service using Reference then you have to register this your injected service, otherwise your test will fail.

4) How Can I test sling model ?

You can use aemContext for that. http://wcm.io/testing/aem-mock/usage.html#Sling_Models


Case 4: Writing test cases for AEM servlets and filters

This is very similar to how you would do test cases for Service. For request and response you either have to mock request / response object using Mockito or Use Spy  or use sling request and response mock. Since a lot of methods in filter and servlet do not return any result, Make Mockito verify your friend.  Here is one example using simple mockito to test servlet



 How can I measure my test coverage ?

 You can use jococo test coverage plugin along with your build system to measure this coverage. You can have following plugin in to your parent pom



How Can I write Integration test in AEM ?

Very good example here https://github.com/sneakybeaky/AEM-Integration-Test-Example

It is based of sling test base https://sling.apache.org/documentation/development/sling-testing-tools.html

I know this information is not enough to have you set up for writing tests in AEM. Feel free to let me know if you have more question and I will add more stuff here.

How to Sightly in AEM / CQ

Use Case: Some common how to in Sightly

Prerequisite: Please read following document carefully before some other How to questions

• Sightly Specification: https://github.com/Adobe-Marketing-Cloud/sightly-spec/blob/master/SPECIFICATION.md
• Basics of Sightly: https://docs.adobe.com/docs/en/aem/6-1/develop/sightly.html
• Sightly tutorials: http://blogs.adobe.com/experiencedelivers/experience-management/sightly-intro-part-1/ (Check other parts on same page)
• Sightly Presentation: http://www.slideshare.net/GabrielWalt/component-development
• Code Base: https://github.com/apache/sling/tree/trunk/bundles/scripting/sightly

Here is some common How to in sightly

• How to include ClientLibs in Sightly 

• How to Loop fixed number of set in Sightly 

• How to create for or while loop in sightly

You can not have dynamic while loop directly in Sightly scripting. However you can use Use class to simulate while or for loop in Sightly. Here is example

• How to reference other OSGI service in sightly 

You can not directly reference OSGI service in sightly template. However you can reference them in use class using getSlingScriptHelper().getService(Service.class) note that you can not use @Reference to refer a service in sightly class.

• How to initialize a default value of a property in sightly

Best way to initialize a value is in activate method of use class. You can also do it in sightly template using something like <sly data-sly-test.parentNavName="${currentPage.properties.navTitle || currentPage.title || currentPage.pageTitle }"> and then use ${parentNavName}

• How to access string array in Sightly 

This is pretty much similar to what we did in for loop. Here is some more generic example

• How to create/access map in sightly 

Here is one example of how you can do that

• How to use Sling Model with Sightly

Here is one example of how you can do it. Note that You can also create sling model for a resource and then use other use classes to return sling model.

• How to hide element in Sightly

You can use either data-sly-unwrap or <sly> tag for this here is example

<!--/* This will only show "Foo" (without a <div> around) if the test is true: */-->
<div data-sly-test="${myTest}" data-sly-unwrap>Foo</div>

<!--/* This would show a <div> around "Foo" only if the test is false: */-->
<div data-sly-unwrap="${myTest}">Foo</div>

<!--/* This will display only the output of the 'header' resource, without the wrapping <sly> tag */-->
<sly data-sly-resource="./header"></sly>

<sly data-sly-unwrap="${false}"></sly> <!--/* outputs: <sly></sly> */-->

• How to pass a sightly object as parameter in Sightly

More use cases to come. Please suggest me any use case you want to get clarity on.

Some more useful Link:

• Creating First Sightly Component: https://helpx.adobe.com/experience-manager/using/creating-sightly-component.html
• Create a AEM project with Sightly Example: https://github.com/Adobe-Marketing-Cloud/aem-project-archetype
• Good Example of Sightly Application using Sling: https://github.com/nateyolles/publick-sling-blog 
• Sightly To do App: https://github.com/Adobe-Marketing-Cloud/aem-sightly-sample-todomvc  
• All Available Context and implicit object in JS and JAVA: http://aemtuts.com/aem-sightly-quick-reference/
• All Available global Object: https://docs.adobe.com/docs/en/aem/6-0/develop/sightly/global-objects.html

Tooling For Sightly:

• Repl Tool: https://github.com/Adobe-Marketing-Cloud/aem-sightly-repl 
• Bracket Extension: https://docs.adobe.com/docs/en/dev-tools/aem-brackets.html 
• Eclipse Extension: https://docs.adobe.com/docs/en/dev-tools/aem-eclipse.html

How to perform Tree Activation in CQ AEM

Use Case: You want to tree activate a page as part of your code or scheduler

Example:

Replication Helper Class

 

Tree Activation Service

Tree Activation Impl



Note: In above example replication action is performed using admin user. You could also create a replication user and use that user to replicate instead.

How to extend Replication Page Process for workflow in AEM

Use Case: You want to extend existing replication page process and add your own logic during workflow step. Some of common use case could be,
1) Activate Asset as soon as it is uploaded to DAM

Solution:

Here is sample code you can use for this,

What this code is doing is trying to find DAM asset path from workflow item and then activating DAM asset instead of activating just original node when OOTB activate page workflow is used.



Once you add your code, you can add this as workflow step in your workflow. If you are extending model.xml for workflow then it will look something like this,

        <node
            jcr:primaryType="cq:WorkflowNode"
            description="A process to activate a page or asset"
            title="Activate Original Asset"
            type="PROCESS">
            <metaData
                jcr:primaryType="nt:unstructured"
                PROCESS="com.wemblog.ActivateAssetFromOriginalProcess"
                PROCESS_AUTO_ADVANCE="true"/>
        </node>

Also you have to make sure that you have right dependency in your pom.xml. If you are using AEM6.1 then you need to use

<dependency>
    <groupId>com.adobe.aem</groupId>
    <artifactId>uber-jar</artifactId>
    <version>6.1.0</version>
    <classifier>apis</classifier>
    <scope>provided</scope>
</dependency>

As usual if you have any question or comment please let me know.

How to set up Shareable Docker Image for Adobe CQ/AEM

Use case

• Large Set up time for developers
• Inconsistent environments across all boxes
• Flexible in container testing
• No Single repository for whole system artifacts 

What is Docker

Docker is a platform for developers and sysadmins to develop, ship, and run applications. Docker lets you quickly assemble applications from components and eliminates the friction that can come when shipping code. Docker lets you get your code tested and deployed into production as fast as possible.
More information about Docker can be found here https://docs.docker.com/
Check Docker Cheat sheet: https://github.com/wsargent/docker-cheat-sheet

Set up

Docker requires two part to work,

• Docker Hub or Docker registry (Where images are stored). You don't have to set up this part. We already have a private docker registry set up for you.
• Docker client (To work with docker images). You need to install this in your local. Please check https://docs.docker.com/installation/ and based on your OS you need to install one. I have instruction to install it on Redhat Linux.

What would you get after following below instruction ?

1. A Private Docker Hub, where you can manage AEM docker package. 
2. A docker Image that will have author, publish and dispatcher instance.
3. Steps to manage your docker hub and local image and make changes to them.

Assumption: I have docker registry set up for Red hat Linux. You might need different steps for different OS. I assume that on your OS mount you have a mount for /export. If this mount is not there then change paths in script.

Docker is only supported FOR RED HAT LINUX VERSION > 6.5 
(Use command cat /etc/*release to find your linux version)

Set Up Private Docker Registry

Prerequisite: 

If you are using redhat then 

Reference:

https://github.com/docker/docker-registry

https://www.digitalocean.com/community/tutorials/how-to-set-up-a-private-docker-registry-on-ubuntu-14-04

Set Up docker Image using docker file

Once docker is install create a dummy directory and put your jar file, license file, Hotfix and other files in to it. More information about Docker file https://docs.docker.com/reference/builder/

Assumption:

1) You have AEM jar file
2) You have License File
3) You have your version of JDK
4) You have compiled HTTPD
5) They all are in same folder where Dockerfile is

For this create a file called Dockerfile and copy below. MAKE CHANGES BASED ON YOUR REQUIREMENT

Once Docker file is there build and create your private docker image using following command. This will create a shareable docker image


Once Docker registry is installed and you have shareable image, now you can install docker client on any other machine

Set up Docker Client

Prerequisite: Based on your OS please install docker client using https://docs.docker.com/installation/

Change default path if there is not enough space

Docker by default get installed under /var/lib/docker in Red Hat. Make sure that you have enough space there. If not then you have to do following.

Set Up docker for Non SSL version

By default Docker uses SSL to communicate to docker registry. If you do not have valid cert installed then you might need to do following


Set up local docker instance using docker client

Do following to start your local AEM instance using docker registry and docker client

How to update docker image

Please follow below step to update your docker image

Troubleshoot
Symptom: You are not able to access your instance from external machine
Solution: Try to restart your machine using 'reboot' for linux box and restart in mac
Symptom: Docker is not starting on mac
Solution: Make sure that you have localhost entry in /etc/hosts file. e.g. 127.0.0.1 localhost. then do following

boot2docker delete boot2docker download boot2docker init boot2docker up # End of this you will be asked to add some env variable in your ~/.bash_profile # Open bash profile and add them

Symptom: Getting following error in mac

FATA[0032] An error occurred trying to connect: Get https://192.168.59.103:2376/v1.17/containers/json: dial tcp 192.168.59.103:2376: i/o timeout

Solution: http://cxwangyi.github.io/notes/2015-01-13-docker-pitfalls.html

boot2docker stop boot2docker destroy boot2docker init boot2docker up docker version

Symptom: On Mac local box not able to access instance using localhost:<port number> (https://docs.docker.com/installation/mac/)
Solution:

# Run following command echo "$(boot2docker ip) localhost" | sudo tee -a /etc/hosts # Then access your instance using localhost # If you want to create other vhost then change above command with your vhost. For example echo "$(boot2docker ip) localhost.wemblog.com" | sudo tee -a /etc/hosts

Some use full docker command

Upload content

You can use Recap tool http://adamcin.net/net.adamcin.recap/ to install content once image is set up. You can even put this recap tool as part of your image when you are creating docker image. Just put them under install folder. 

Note that every time you update your image, Uploaded content or local changes might get overriden. You have to commit your changes using docker commit and then apply update on top of that and then share your image.

Useful resource: http://www.techinsight.io/review/adobe-experience-manager/incontainer-testing-for-aem-with-docker/

Note: As usual let me know if you have any question.

How to Set Up Clustering In CQ/AEM 6 using MongoDB

Background:

With CQ / AEM 6 TarPM is not supported any more. AEM 6 ships with Oak which for now support TarMK and MongoMK Microkernal OOTB. More information about what is New Can be found from http://www.slideshare.net/AEMHub2014/oak-michael-marth . With this change Support from Clustering is moved to actual storage layer it self (Which make more sense, given supporting all issues for clustering in earlier version). TarMK does not have replication or sharding feature so it comes down to MongoDB which support replication and sharding and hence enable High Availability (HA through replication) and Scalability (Through Sharding, Though this is still a question ?? See note below) through clustering in CQ /AEM 6.

Here we will give step by step instruction of how to set up clustering using MongoDB in CQ

Pre requisite:

• Make sure that You have CQ / AEM 6 jar file
• Install MongoDB using instruction http://docs.mongodb.org/manual/installation/
• Make sure that it is up and running
• Read about replication http://docs.mongodb.org/manual/core/replication-introduction/
• Read about deployment Strategy http://docs.adobe.com/docs/en/aem/6-0/deploy/recommended-deploys.html

There are two cases for setting Up Replica Set:

Set up a new MongoDB Instance:

• Set up additional MongoDB instance based on instruction above
• Start any one of instance using ./mongod --port <Your Port> --dbpath <Your DB Path> --replSet <Replica Set Name could be any thing> &  
• You can also use configuration file to do that. More instruction here http://docs.mongodb.org/manual/tutorial/deploy-replica-set/
• Once Mongo DB is started you can add additional replica using following instruction 

• Once Replica set is up, Now set Up AEM

• Then You can go to each Mongo Instance and check of data is coming using Mongo Log

Convert Existing Mongo Instance:

• Stop you AEM instance
• Use Following instruction to convert Mongo to replica

• Once this is set Change AEM start script to add mongo replica instance as given in approach one 
• start your AEM instance
• AEM should be part of replica set now

Backup and Restore

Please check https://docs.mongodb.org/v3.0/tutorial/backup-and-restore-tools/ for MongoDB instruction of backup and restore. 

Automated script can be found here: https://github.com/micahwedemeyer/automongobackup/blob/master/src/automongobackup.sh just put this script under /etc/cron.daily and you are set for backup.

Some Common Questions

Should I set up my AEM author instance on MongoDB

Unless you have clustering requirement, I would not suggest to set up your author instance with MongoDB. Mainly because of administrative overhead.

Should I set up my AEM publish instance on MongoDB

Same as above, Unless you have a requirement which requires shared content generation I would suggest not to use MongoDB. With AEM communities, now you have an option to add Mongo Persistence for community feature at any time. Here is more detail https://docs.adobe.com/docs/en/aem/6-1/administer/communities/srp/msrp.html and https://docs.adobe.com/docs/en/aem/6-1/administer/communities/srp/msrp/demo-mongo.html

Should I store Blobs in MongoDB as well in AEM

It is not recommended to store Blob data with MongoDB. There are other options like, Local Storage, NAS, AWS you can use in that case. More detail https://docs.adobe.com/content/docs/en/aem/6-1/deploy/platform/aem-with-mongodb.html#AEM Configuration and https://docs.adobe.com/docs/en/aem/6-1/deploy/platform/data-store-config.html

How can I secure my MongoDB deployment with AEM

Check documentation here https://blogs.adobe.com/security/2015/07/securely-deploying-mongodb-3-0.html

Notes:

1) Mongo Replication Only Provide High Availability (HA) it does not provide scalability. For scalability you need to use Sharding feature provided by Mongo. However I am not sure what would be best key to create shard on for Mongo. You can create Shard based on _id attribute. More information about sharding can be obtained here http://docs.mongodb.org/manual/sharding/  . If you are using Sharding I would suggest to use sharding with replication (Shard and then replicate shard instance) to provide both HA and scalability.

2)  There are many feature available in Mongo Replication where you can make certain replica instance read only (Data Center replica), you can use this to avoid high latency across Data Center here is all configuration you can do on Mongo http://docs.mongodb.org/manual/administration/replica-set-member-configuration/

3) MongoDB recently released MMS https://mms.mongodb.com/ to monitor and deploy Mongo Cluster easily. This will be useful if you are worried about administrative cost for Mongo 

4) If you don't want to store large documents in Mongo feel free to use custom Data Store using instruction here http://jackrabbit.apache.org/oak/docs/osgi_config.html

5) Mongo Recently launched another feature of pluggable datastore. You can use this for faster read and write based on your requirement (For example Primary with high Write Enabled Storage Like SSD or something and read with cheap storage). More info here https://www.mongosoup.de/blog-entry/A-closer-look-at-pluggable-storage.html (Official Doc yet to come)

6) Official AEM Documentation: https://docs.adobe.com/content/docs/en/aem/6-1/deploy/platform/aem-with-mongodb.html

Finally .... Some more Mongo Command ...

Special Thanks To Nelson Mei for Setting up POC for Mongo with AEM

How to use impression Service In CQ/AEM

Use Case:

1. You often have case where you want to use Impression service provided by CQ to do custom operation for example finding top 10 most viewed page or sorting all page based on there popularity.
2. It might possible that your impression data (Page Views) is in external system and then you want to import those data as impression in CQ to have more application context.
3. You want to aggregate all data across all publish instances.

Solutions:

Approach 1:

Creating your Own Impression service

You can create your own impression service by extending com.day.crx.statistics.Entry here is example

Supporting class

You need to embed following dependency for this
Here is example of how you can use this service

Now you can import data from external system (GA, Site Catalyst, Kafka) and then populate it using this service to your instance.

Once you are ready with all data you can use following service to use data,



Actual Implementation


Approach 2:

You don't want to write your own service as mentioned in Approach 1 and use OOTB service available to you. Only problem with this is, You have multiple publish instance and some how you want to combine all data in to one so that you get accurate picture. It kind of tricky to get all data from all publish instance (through reverse replication) and then combine them on author and then push them over again. However you can use one instance to collect all stat data (king of single source of truth and then replicate it back to all instance every day)

• Make sure that you enable page view tracking by adding following line

           <cq:include script="/libs/foundation/components/page/stats.jsp" />

• Then configure all publish instance to point to one DNS using following config (You can always override this under /apps)

           /apps/wcm/core/config.publish/com.day.cq.wcm.core.stats.PageViewStatistics

          /apps/wcm/core/config.publish/com.day.cq.wcm.core.stats.PageViewStatisticsImpl

         

• make sure that pageviewstatistics.trackingurl is pointing to single domain (You need to create a domain, something like impression.mydomain.com that will be stand alone CQ instance to take all impression request)
• Now you have consolidated page impression on one machine
• You can easily write a schedular which will run every night and reverse replicate all data to author instance.
• Once it is on author instance you can use replicator service to replicate to all other publish instance
• Then you can use code mention in approach 1 to get popular resources.

Note: You can always use GA or something to track data. This is more useful if you want to do something internally and not want top share data with GA.

As usual feel free to ask any question you have.

How to make Instances SSL context aware in CQ/AEM

Use Case:

Most of the times we terminate SSL on load balancer or at dispatcher and communication to publish happen over http. In this case publish server is often not SSL aware and any request specific operations (For example relative path redirect or Link rewriting) happens over http. For example if you do something like request.sendRedirect("/somepath") from server it will get redirected to http://server-name/somepath or when you will do request.isSecure() it will return false. Operations like externalizer.externalLink(resolver, "mydomain", "/my/page") + ".html"; will also return http version of link.

Solution:

Option 1:

Let all links be http and then do force redirect on dispatcher or Load Balancer. For dispatcher rule can be as simple as this,

RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [L,R=301]

Issue:

• Not SEO friendly (Unless it is relative link).  
• Can not embed as portlet or Iframe (Cross Site include error) over https.

Option 2: 

Use SSL filter from Felix HTTP services http://felix.apache.org/documentation/subprojects/apache-felix-http-service.html#using-the-ssl-filter

If filter is configured, it looks for header configured in configuration and if header value matches configured value context on publish is considered secure and request.isSecure() will return true.

In above case if X-Forwarded-SSL is present with value on then servlet context is secure.

Note: This might not work in version latest CQ version. You can build your own version from trunk https://github.com/apache/felix/tree/trunk/http/sslfilter or from http://svn.apache.org/repos/asf/felix/trunk/http/sslfilter/

One working version of file can be downloaded from here

As usual let me know if you have any question. Special thanks to Shenghao Huang from LinkedIn for finding this.

How to use Sling Models in CQ5.6

Use Case: Use Sling Model in CQ5.6

Background: Sling model http://sling.apache.org/documentation/bundles/models.html brings powerful way of mapping your resources to Java Objects (Beans) . It is supported OOTB in CQ6 however we can use it in CQ5.6 as well.

Solution:

First add sling models as dependency in your code (In Parent pom using dependency manager is preferred)

Then update your reactor pom (Which actually creates your CQ package using content-package-maven-plugin) you need to add following as dependency and then embed them in to your project (If install path for system is not present you can use any other path you want)

Thats it .. Models are available as service in your OSGI environment.

 To use model go ahead and create your own bundle and them add these as dependency. You should be able to write, deploy and run code against model. Note that you might have to make these model available to osgi using Sling-Model-Packages



Once model is deployed correctly you should be able to see them under sling-model tab in status in felix console or by going to HOST:PORT/system/console/status-slingmodels

Some example code using models

Some common How to

How can I inject services in Sling Model

by using annotation @OSGIService Or Using Injector @Inject @Source("osgi-service") 

Example:

@OSGIService
MyService myService

@Inject @Source("osgi-service")
MyService myService

How Can I Inject Sling Object in Sling Model

by using annotation @SlingObject Or Using Injector @Inject @Source("sling-object") 

Example:

@SlingObject
ResourceResolver resourceResolver

@Inject @Source("sling-object")
ResourceResolver resourceResolver

How Can I get access to property if adapting through Sling Request

If you are adapting your model to sling request you might not have access to property by just doing @Inject. You have to use @Via in that case. Note that you can use all other annotation here like @Named and all. Here is example

@Model(adaptables=SlingHttpServletRequest.class)
public interface MyModel { 

    @Inject @Via("resource")
    String propertyName;
}

How Can I use Sling Tag Library with Sling Model

You can use <sling:adaptTo ... /> or ${sling:adaptTo ...} as mentioned in https://sling.apache.org/documentation/bundles/models.html and https://sling.apache.org/documentation/bundles/sling-scripting-jsp-taglib.html .In 5.6 you might have to change global.jsp to use
<%@taglib prefix="sling" uri="http://sling.apache.org/taglibs/sling" %>
instead of
<%@taglib prefix="sling" uri="http://sling.apache.org/taglibs/sling/1.0" %>

How Can I provide default value to property in Sling Model

By using @Default Annotation . Here is example

A default value can be provided (for Strings & primitives) and Array:

@Model(adaptables=Resource.class)
public class MyModel {

    @Inject @Default(values="defaultValue")
    private String name; 

    @Inject @Default(intValues={1,2,3,4})
    private int[] integers; 

}

How Can I inject child resource as Model in Sling Model 

This can be done by Injecting a model in other model class. Here is example

@Model(adaptables=Resource.class)
public interface MyModel {

    @Inject
    ImageModel getImage();
}

@Model(adaptables=Resource.class)
public interface ImageModel {

    @Inject
    String getPath();
}

When a resource is adapted to MyModel, a child resource named image is automatically adapted to an instance of ImageModel

Note: Sling model version number can change as it evolves in future. Make sure that you update model dependencies version accordingly. You can also use Sling testing framework to test sling model which is pretty cool as well. There is also an example of deployable Model package https://github.com/Adobe-Consulting-Services/com.adobe.acs.bundles.sling-models which you can use.

As usual feel free to give your comment and feedback and let me know if you want me to add something else here.

How to Connect to External Data Source (Teradata / MySQL / Oracle) in CQ / AEM

Use Case: Connect to external datasource using CQ

Solution:

1) My SQL

Step 1: Create OSGI version of mysql Jar file

• Download MySQL JDBC Driver From http://dev.mysql.com/downloads/connector/j 
• Open eclipse and select file -> New -> Other -> Plugin development -> Plug-in from existing Jar Archive

• Click next and then select add external. Select jar file you downloaded above and select next
•  Give Project Name -> Select Location -> Make sure that Analyze Library Content is checked -> In Target Platform select an OSGI framework -> from drop down select standard -> Check unzip jar file and update reference -> click finish

• An Plugin Development prospective will open 
• Click on export tab and make sure that all dependencies are exported (If they are not there, Click on add then select all)

• Click on Dependencies and make sure that Imported packages are there. If it is not there click on MENIFEST.MF tab and then add following import statement 
• Once all import and export statement is added your MENIFEST.MF will have following data

• Then right click on your project and then select export 
• Select Deployable Deployable plug-in and fragments

• Click Next and select your plug-in 
• From Directory section select location where you want to upload.
• Click finish. This will store OSGI jar in to location you selected.
• Once you have MySQL Osgi bundle. Install that bundle either using felix console or adding it through deployment process

• If you are getting error saying javax.Naming not getting resolved then you have to add following line in sling.properties under crx-quickstart folder

org.osgi.framework.bootdelegation= javax.naming.*, ${org.apache.sling.launcher.bootdelegation}

• Once bundle is up is running, You need to go to felix osgi config. Search For JDBC connection pool and then click '+'
• Fill all the information about your datasource

• Go to your application logic and test your code

If you are having trouble creating osgi version of mysql jar file, You can download it from here

2) Teradata

• All the steps for Teradata will remain same except you need to create teradata OSGI bundle using Teradata JDBC connector.
• First Download teradatajdbc connector from here https://code.google.com/p/kyunra-java/downloads/detail?name=terajdbc4.jar
• Then Download tdconfig jar from here https://code.google.com/p/kyunra-java/downloads/detail?name=tdgssconfig.jar
• Follow above steps to create osgi version of jar file (Everything will remain same except this time you need to select two jar file for creating osgi version of it)
• Your Final MENIFEST.MF will look like this

• Upload final jar file in CQ, make sure that it is active

• Open OSGI config to create config for teradata connection pool

• You can then test it using same code (Just use teradata data source)

You can also download OSGI version of teradata jar file from here

Note: If You want to call instance of your Connection from Java class (Not from service) You can do something like this,

How to Use Sessions and Resource Resolver through Service Authentication In AEM6

Use Case: As per http://sling.apache.org/documentation/the-sling-engine/service-authentication.html and http://docs.adobe.com/content/docs/en/aem/6-0/develop/ref/diff-previous/changes/changes-summary.html using admin session and admin resource resolver through ResourceresolverFactory is now deprecated. Using Service based Authentication for Resourceresolver and Respository session solves problem like (Directly From Sling Doc),

• Prevent over-use and abuse of administrative ResourceResolvers and/or JCR Sessions
• Allow services access to ResourceResolvers and/or JCR Sessions without requiring to hard-code or configure passwords
• Allow services to use service users which have been specially configured for service level access (as is usually done on unixish systems)
• Allow administrators to configure the assignment of service users to services

Solution:

NOTE: Use Service Accounts for alice and bob users [jcr:primaryType=rep:SystemUser] instead of regular accounts.

Lets see we have two user "alice" and "bob", with following property,

• "alice" only have READ access to document under /content/somepath path
• "bob" has both read and write access to document under /content/somepath path

Now we have two service "ReadService" and "WriteService", with following property

• ReadService should only be allowed to read anything under /content/somepath path
• WriteService should be allowed for both read and write under /content/somepath path

Assume your package name is blog.wemblog.com

Step 1: Create ReadService and WriteService using resourceResolver Or adminSession using new Authentication Service based API

Step 2: Create ReadService same way
Step 3: Update org.apache.sling.serviceusermapping.impl.ServiceUserMapperImpl config by creating osgi:config node under /apps/<your-apps>/config.<Place where you want to run this>/org.apache.sling.serviceusermapping.impl.ServiceUserMapperImpl.xml you can directly go to osgi config through Felix console and change this as well look for  “Apache Sling Service User Mapper Service” for that.

Syntax for service mapping to user is ‘serviceName [ ":" subServiceName ] “=” username’.

and Entry of OSGI config will look like this,

After installing the bundle and configuration and code, You would see something like this in log

*INFO*  blog.wemblog.com.ReadServiceImpl alice

*INFO*  blog.wemblog.com.ReadServiceImpl <node type of somepath>

*INFO*  blog.wemblog.com.WriteServiceImpl bob

*INFO*  blog.wemblog.com.WriteServiceImpl <node type of somepath>

*INFO*  blog.wemblog.com.WriteServiceImpl Successfully saved

If you need to use admin session for the configuration you can do something like blog.wemblog.com:WriteService=admin in osgi config above. Good practice is to have these session based on groups depending upon which group have access to what service.

You might need following dependencies in your POM for api to be available

Please check http://stackoverflow.com/questions/31350548/resourceresolverfactory-getserviceresourceresolver-throws-exception-in-aem-6-1 with some of the changes in AEM6.1 of how to use this.

As always feel free to ask any question you might have.