Pass arguments to your main method in gradle bootRun

If you are tired of Spring Boot configuration magic, there is one more trick to confuse you completely.
How to configure Spring to work without knowing profile or environment? Dropwizard style, like this:
java -jar myjar.jar myconfig.yml.
No profiles, no wandering how did my properties got populated.

Spring Boot first class configuration is Java bean. Unfortunately, due to legacy issues, we need to include xml config. After some pain, suffering and reading, I found solution:

@ImportResource("classpath:application-context/applicationContext.xml")
//@Component
public class BatchConfiguration 

In similar way you can include properties file.
Next step is ${properties}. In xml configuration there are some environment-specific properties, like database url and so on. Some advise JNDI, but it introduces one more layer of magic - configuration of a web container.
So, question is, how to populate properties with external config. There are a ton of solutions out there, but none worked for me. Maybe because it's Spring Boot+Spring Data+Spring Batch or maybe because I don't understand this page. Anyway, I found my own way:

    
public static void main(String[] args) throws IOException {
        if (args.length >= 1) {
            Properties p = new Properties();
            p.load(new FileReader(args[0]));
            p.forEach((x, y) -> {
                System.setProperty((String) x, (String) y);
            });
        }
        SpringApplication.run(Application.class, args);
    }

Now it works perfectly in fatJar task, but what about bootRun? Boot Run saves a lot of dev time and is very easy to use. As you probably know, BootRun extends gradle standard JavaExec task, with awesome parameter args. It goes straight to your main method. So gradle bootRun task looks like:

bootRun {
    args =["cars-etl.properties"]
    
}

This is all for make your solution work! Have fun.
P.S. HowTo pass jvmArgs
P.P.S. HowTo run job from Controller

MongoBulkItemWriter

Spring Batch is a nice choice for simple ETL jobs, but it doesn't work well with mongodb, especially writing to it. Provided in Spring Batch MongoItemWriter doesn't do bulk inserts.
Fortunately for us, bulk inserts are quite easy to implement:

import com.mongodb.BulkWriteOperation;
import com.mongodb.BulkWriteResult;
import com.mongodb.DBObject;
import org.springframework.batch.item.ItemWriter;
import org.springframework.data.mongodb.core.MongoTemplate;

import java.util.List;

public class MongoBulkItemWriter<T> implements ItemWriter<T> {

    private String collection;
    private MongoTemplate template;

    public MongoBulkItemWriter(String collection, MongoTemplate mongoTemplate) {
        this.collection = collection;
        this.template = mongoTemplate;
    }

    @Override
    public void write(List items) throws Exception {
        BulkWriteOperation bulk = template.getCollection(collection).initializeUnorderedBulkOperation();
        items.forEach(i->{
                bulk.insert((DBObject) template.getConverter().convertToMongoType(i));
        });
        BulkWriteResult result = bulk.execute();
    }
}

It works much faster, but beware - inserts only, so item with duplicate id will ruin your batch. Solution might look something like this:

        BulkWriteOperation bulk = template.getCollection(COLLECTION_NAME).initializeUnorderedBulkOperation();
        updates.forEach(u -> {
            bulk.find(new BasicDBObject("id", u.getId())).upsert().update(u.getDbObject());
        });
        bulk.execute();

Upserts are much slower than pure inserts, but still a huge win compared with per object writes.

Spring Data Mongo Criteria Example

spring-data-mongodb-1.8.4 has a great package , with some features that makes mongodb queries easy. Unfortunately, documentation on this topic is not very good. I'll contribute to it by short example.

Problem

We want to search mongodb using $and and $in keyword based on several fields. But we don't know field configuration beforehead. Input params are fit into domain class:

public class MakeModelYear {
    private String[] make;
    private String[] model;
    private String[] year;

    public MakeModelYear(String[] make, String[] model, String[] year) {
        this.make = make;
        this.model = model;
        this.year = year;
    }

    public String[] getMake() {
        return make;
    }

    public String[] getModel() {
        return model;
    }

    public String[] getYear() {
        return year;
    }

    public boolean isEmpty() {
        return isEmpty(model) && isEmpty(make) && isEmpty(year);
    }

    public static boolean isEmpty(String[] a) {
        return a == null || a.length ==0;
    }
}

Any of make model and year can be null, and we still need to have nice smooth functionality

Solution

For Criteria object:

   public static Criteria generateQuery(MakeModelYear key) {
        Queue cril = new ArrayDeque<>();
        if (!MakeModelYear.isEmpty(key.getMake())) {
            Criteria makeCri = Criteria.where("Make").in(Arrays.asList(key.getMake()));
            cril.add(makeCri);
        }
        if (!MakeModelYear.isEmpty(key.getModel())) {
            Criteria modelCri = Criteria.where("Model").in(Arrays.asList(key.getModel()));
            cril.add(modelCri);
        }
        if (!MakeModelYear.isEmpty(key.getYear())) {
            Criteria yearCri = Criteria.where("Year").in(Arrays.asList(key.getYear()));
            cril.add(yearCri);
        }

        return cril.size() == 1 ? cril.poll() : cril.poll().andOperator(cril.toArray(new Criteria[]{}));
    }

For Query itself object:

        MakeModelYear key = new MakeModelYear(make, model, year);
        Query query = new Query();
        if(!key.isEmpty())query.addCriteria(DaoHelper.generateQuery(key));
        query.fields().exclude("_id");
        return mongoTemplate.getCollection(CARS).find(query.getQueryObject(), query.getFieldsObject()).toArray();

As you can see, we can specify both Criteria and configuration what to return and what to supress in one readable object.
I could not find better way in Jongo and mongo driver.