MongoBulkItemWriter

Spring Batch is a nice choice for simple ETL jobs, but it doesn't work well with mongodb, especially writing to it. Provided in Spring Batch MongoItemWriter doesn't do bulk inserts.
Fortunately for us, bulk inserts are quite easy to implement:

import com.mongodb.BulkWriteOperation;
import com.mongodb.BulkWriteResult;
import com.mongodb.DBObject;
import org.springframework.batch.item.ItemWriter;
import org.springframework.data.mongodb.core.MongoTemplate;

import java.util.List;

public class MongoBulkItemWriter<T> implements ItemWriter<T> {

    private String collection;
    private MongoTemplate template;

    public MongoBulkItemWriter(String collection, MongoTemplate mongoTemplate) {
        this.collection = collection;
        this.template = mongoTemplate;
    }

    @Override
    public void write(List items) throws Exception {
        BulkWriteOperation bulk = template.getCollection(collection).initializeUnorderedBulkOperation();
        items.forEach(i->{
                bulk.insert((DBObject) template.getConverter().convertToMongoType(i));
        });
        BulkWriteResult result = bulk.execute();
    }
}

It works much faster, but beware - inserts only, so item with duplicate id will ruin your batch. Solution might look something like this:

        BulkWriteOperation bulk = template.getCollection(COLLECTION_NAME).initializeUnorderedBulkOperation();
        updates.forEach(u -> {
            bulk.find(new BasicDBObject("id", u.getId())).upsert().update(u.getDbObject());
        });
        bulk.execute();

Upserts are much slower than pure inserts, but still a huge win compared with per object writes.

Spring Data Mongo Criteria Example

spring-data-mongodb-1.8.4 has a great package , with some features that makes mongodb queries easy. Unfortunately, documentation on this topic is not very good. I'll contribute to it by short example.

Problem

We want to search mongodb using $and and $in keyword based on several fields. But we don't know field configuration beforehead. Input params are fit into domain class:

public class MakeModelYear {
    private String[] make;
    private String[] model;
    private String[] year;

    public MakeModelYear(String[] make, String[] model, String[] year) {
        this.make = make;
        this.model = model;
        this.year = year;
    }

    public String[] getMake() {
        return make;
    }

    public String[] getModel() {
        return model;
    }

    public String[] getYear() {
        return year;
    }

    public boolean isEmpty() {
        return isEmpty(model) && isEmpty(make) && isEmpty(year);
    }

    public static boolean isEmpty(String[] a) {
        return a == null || a.length ==0;
    }
}

Any of make model and year can be null, and we still need to have nice smooth functionality

Solution

For Criteria object:

   public static Criteria generateQuery(MakeModelYear key) {
        Queue cril = new ArrayDeque<>();
        if (!MakeModelYear.isEmpty(key.getMake())) {
            Criteria makeCri = Criteria.where("Make").in(Arrays.asList(key.getMake()));
            cril.add(makeCri);
        }
        if (!MakeModelYear.isEmpty(key.getModel())) {
            Criteria modelCri = Criteria.where("Model").in(Arrays.asList(key.getModel()));
            cril.add(modelCri);
        }
        if (!MakeModelYear.isEmpty(key.getYear())) {
            Criteria yearCri = Criteria.where("Year").in(Arrays.asList(key.getYear()));
            cril.add(yearCri);
        }

        return cril.size() == 1 ? cril.poll() : cril.poll().andOperator(cril.toArray(new Criteria[]{}));
    }

For Query itself object:

        MakeModelYear key = new MakeModelYear(make, model, year);
        Query query = new Query();
        if(!key.isEmpty())query.addCriteria(DaoHelper.generateQuery(key));
        query.fields().exclude("_id");
        return mongoTemplate.getCollection(CARS).find(query.getQueryObject(), query.getFieldsObject()).toArray();

As you can see, we can specify both Criteria and configuration what to return and what to supress in one readable object.
I could not find better way in Jongo and mongo driver.