MongoBulkItemWriter

Spring Batch is a nice choice for simple ETL jobs, but it doesn't work well with mongodb, especially writing to it. Provided in Spring Batch MongoItemWriter doesn't do bulk inserts.
Fortunately for us, bulk inserts are quite easy to implement:

import com.mongodb.BulkWriteOperation;
import com.mongodb.BulkWriteResult;
import com.mongodb.DBObject;
import org.springframework.batch.item.ItemWriter;
import org.springframework.data.mongodb.core.MongoTemplate;

import java.util.List;

public class MongoBulkItemWriter<T> implements ItemWriter<T> {

    private String collection;
    private MongoTemplate template;

    public MongoBulkItemWriter(String collection, MongoTemplate mongoTemplate) {
        this.collection = collection;
        this.template = mongoTemplate;
    }

    @Override
    public void write(List items) throws Exception {
        BulkWriteOperation bulk = template.getCollection(collection).initializeUnorderedBulkOperation();
        items.forEach(i->{
                bulk.insert((DBObject) template.getConverter().convertToMongoType(i));
        });
        BulkWriteResult result = bulk.execute();
    }
}

It works much faster, but beware - inserts only, so item with duplicate id will ruin your batch. Solution might look something like this:

        BulkWriteOperation bulk = template.getCollection(COLLECTION_NAME).initializeUnorderedBulkOperation();
        updates.forEach(u -> {
            bulk.find(new BasicDBObject("id", u.getId())).upsert().update(u.getDbObject());
        });
        bulk.execute();

Upserts are much slower than pure inserts, but still a huge win compared with per object writes.

Spring Data Mongo Criteria Example

spring-data-mongodb-1.8.4 has a great package , with some features that makes mongodb queries easy. Unfortunately, documentation on this topic is not very good. I'll contribute to it by short example.

Problem

We want to search mongodb using $and and $in keyword based on several fields. But we don't know field configuration beforehead. Input params are fit into domain class:

public class MakeModelYear {
    private String[] make;
    private String[] model;
    private String[] year;

    public MakeModelYear(String[] make, String[] model, String[] year) {
        this.make = make;
        this.model = model;
        this.year = year;
    }

    public String[] getMake() {
        return make;
    }

    public String[] getModel() {
        return model;
    }

    public String[] getYear() {
        return year;
    }

    public boolean isEmpty() {
        return isEmpty(model) && isEmpty(make) && isEmpty(year);
    }

    public static boolean isEmpty(String[] a) {
        return a == null || a.length ==0;
    }
}

Any of make model and year can be null, and we still need to have nice smooth functionality

Solution

For Criteria object:

   public static Criteria generateQuery(MakeModelYear key) {
        Queue cril = new ArrayDeque<>();
        if (!MakeModelYear.isEmpty(key.getMake())) {
            Criteria makeCri = Criteria.where("Make").in(Arrays.asList(key.getMake()));
            cril.add(makeCri);
        }
        if (!MakeModelYear.isEmpty(key.getModel())) {
            Criteria modelCri = Criteria.where("Model").in(Arrays.asList(key.getModel()));
            cril.add(modelCri);
        }
        if (!MakeModelYear.isEmpty(key.getYear())) {
            Criteria yearCri = Criteria.where("Year").in(Arrays.asList(key.getYear()));
            cril.add(yearCri);
        }

        return cril.size() == 1 ? cril.poll() : cril.poll().andOperator(cril.toArray(new Criteria[]{}));
    }

For Query itself object:

        MakeModelYear key = new MakeModelYear(make, model, year);
        Query query = new Query();
        if(!key.isEmpty())query.addCriteria(DaoHelper.generateQuery(key));
        query.fields().exclude("_id");
        return mongoTemplate.getCollection(CARS).find(query.getQueryObject(), query.getFieldsObject()).toArray();

As you can see, we can specify both Criteria and configuration what to return and what to supress in one readable object.
I could not find better way in Jongo and mongo driver.

Hexagonal architecture and ESB (Enterprise service bus). Part 1.

This blog post will start comparing 2 architectural approaches. 

You probably have heard about ESB - architectural model to connect everything with everything. It's interesting for me to compare it to less known but quite interesting concept of Hexagonal architecture.
In enterprise there are usually a tons of interconnected services\applications, so let's evaluate these approaches from enterprise view.

Overview 

In a one sentence:

Hexagonal architecture(HA) - ‘’every’’ piece of functionality the application(service) offers is available through an API (application programmed interface) or function call.

Enterprise service bus(ESB) - mediator service that facilitates work of loosely coupled applications(services) that are expected to be independently deployed, running, heterogeneous, and disparate within a network.
 Nobody reads middle section, so I will leave it for next posts and jump directly to a conclusion.

Conclusion

Hexagonal architecture might seem less mature, but, compared to enterprise service bus, it provides a lot of practical benefits for multiple services integration. They might look uglier on a paper, but, in practice, they provide faster time-to-market software with better quality and simpler error handling.
Good news is, that HA is compatible with ESB, so I think best way is to use HA as a basic approach, adding some ESB as you need it.
In one sentence - design your application around your business domain, not integration solution.

Dropwizard vs Spring Boot

In Java world there are 2 most wide spread solutions for JSON REST technology. In this post I’ll cover some aspects of them, that seems of most importance to me. 

Dependency Injection

With Spring, you are limited to Spring DI - quite complex and hard to debug. With Dropwizard, you can choose any DI you like. I was using Guice. But beware of using javax.inject annotations. Dropwizard v0.9.2. uses fresh release Jetty, that was build using HK2 DI library. So, using  general annotations with some other DI solution may result in strange behavior. For Guice-HK2 there are some bridge solutions, but using DI library specific annotations will save you from trouble.

Rest Api

I personally find JAX-RS specification not very convenient, maybe because I prefer single routes file and dislike annotating classes with @Path. Spring annotations for REST API are slightly better in this regard.

Configuration

Dropwizard offers once convenient way to configure everything - yml. With Spring you have as much flexibility as you can handle, and it makes server hard to maintain. I guess Dropwizard configuration is more in line with java values - there is single way to do it, simple, clear,maintainable. In case of issue you always know where to look.

Deployment

Dropwizard oriented on jar-deployment, standalone jetty. Spring can do both jar and war (for tomcat). It is arguable, but I consider multiple deployment options as a plus - it doesn’t complicate maintenance much, but can simplify support in case of running multiple applications.

Static Resources & Templates

Dropwizard offers quite simple way to mix pages and JSON API. If we will use angular on client side - it result in quite elegant solution with no tweaks. With Spring Boot I've struggled to configure my application to work with HTML and JSON, with support of some static ( e.g. css) resources.

SummaryTable

This is my marks for different aspects of platform. Max mark is 5. The bigger-the better.

Criteria	Dropwizard	Spring Boot
DI	5	3
REST API	3	4
Configuration	5	3
Deployment	4	5
Static Resources & Templates	5	3
Total	22	18

Summary

Dropwizard is better IMHO, and it plays nice with most Spring projects, like Spring Security and Spring JDBC. Spring Boot probably more performant, but I have no data on this topic, and all performance depends mostly from performance of your database. Solutions are quite similar, so winner detected.

There are lots of other articles on topic, that cover slightly different areas than this one.

See http://blog.takipi.com/java-bootstrap-dropwizard-vs-spring-boot/  and http://www.schibsted.pl/2015/07/spring-boot-and-dropwizard-in-microservices-development/ for more.

OOP - we are fooling ourselves

Object Oriented Programming is a holy grail of modern code-writing methodologies. As everything so fundamental, it was criticised  quite heavily, with different degree of details. Some say, OOP is dead, some say it’s harmful. I think, we should reconsider our understanding of this paradigm.

If we talk about objects - we think about state (a lot) and behaviour (a little) . But in all domain models  we separate between «domain» and «business» logic. 

Example: A human buys bananas in a grocery store. Here set of objects Bananas change ownership and location. Object Store obtains money, frees some space in a shop, and loose some bananas. Object human looses money and gets a bananas. 

Question: What object should own this logic? If it’s 3 different objects, each with it’s own functions, then who should coordinate behaviour? Is it a shop who should take money from a human and change ownership of bananas? Or Human? Or Banana? Or some different different Object?

In practice, usually it’s different object, most probably StoreService. 

After that we have another question - about persistence. Somebody should store it to a database (persistent storage). Should it be domain objects? In practice, it’s usually DAO layer. 

And I will not dig into details - because human usually need to pack bananas in a bag and store should manage not to run out of bananas.

So, what do we have here - our objects are passive data objects, and all behaviour is in other objects, with some dependency on external resources (database).

Is it OOP ? Well, in practice, it’s what everybody calls OOP. Clearly, something is wrong.

Looking on a Store, we will not have an idea about all it’s abilities. StoreService will give us much more information. 

There is one solution I know to this problem - mix-ins. You will be able to mix some resource-dependent behaviour into domain objects, without altering domain objects much. Is it OOP ?

Nice articles on a subject:

http://loup-vaillant.fr/articles/deaths-of-oop

http://www.smashcompany.com/technology/object-oriented-programming-is-an-expensive-disaster-which-must-end

Hazelcast trick 1

I was playing with hazelcast for a while. Great tool for java-based distributed systems - simple and elegant.
During our work I've run into some tricky situations. I understand that they can be solved in different ways, I've shosen to write some bits of code. I'll share these bits with explanations in upcoming posts.

Hazelcast Trick 1. Multicast vs TcpIpJoiner

Problem: 

In our network some clusters were in several subnets - and I've not figured out how to fix this yet, but hazelcast has tcp-ip joiner to the resque. There is one disadwantage though. Same config file for different machines makes it hard not to include localhost in config. And hazelcast was not working with hostnames correctly for me.

Solution:

So, I've decided to keep config file uniform, but add following code to my startup procedure:

       Config cfg = new FileSystemXmlConfig(configFile);
        cfg.setInstanceName(configName);
        if (tcpipjoin != null && tcpipjoin.isEnabled()) {
            cfg.getNetworkConfig().getJoin().getTcpIpConfig().clear();
            cfg.getNetworkConfig().getJoin().getTcpIpConfig().setEnabled(true);
            cfg.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
            for (String host : tcpipjoin.getHosts()) {
                String resolvedHost = resolveHostToIp(host);
                if (resolvedHost != null) {
                    cfg.getNetworkConfig().getJoin().getTcpIpConfig().addMember(resolvedHost);
                }
            }
        }

This solved 2 issues:

1. Resolve hostname to IP (oddly, hazelcast was not doing well with hostnames, dont know why yet)
2. Exclude localhost from list - in this final configuration was simple and effective. Without attempting to call itself and rezolve hostname.

Helper methos is not as straitforward, here it goes:

    static String resolveHostToIp(String host) throws UnknownHostException {
        String[] parts = host.split(":");
        if (parts.length > 2) throw new UnknownHostException("Illegal host format: " + host);
        String ip = InetAddress.getByName(parts[0].trim()).getHostAddress();
        if(isLocal(ip)) {
            logger.info("Host " + host + "filtered out as local");
            return null;
        }
        String resolved = parts.length == 1 ? ip : (ip + ":" + parts[1].trim());
        logger.debug("host " + host + " resolved to " + resolved);
        return resolved;
    }

    private volatile static List localhostIpSynonyms;

    private static boolean isLocal(String ip) {
        if (localhostIpSynonyms != null) {
            return localhostIpSynonyms.contains(ip);
        }
        else {
            try {
                List synonims = new ArrayList<>();
                for (Enumeration ifcs = NetworkInterface.getNetworkInterfaces(); ifcs.hasMoreElements(); ) {
                    NetworkInterface ifc = ifcs.nextElement();
                    if (ifc.isUp()) {
                        for (Enumeration ias = ifc.getInetAddresses(); ias.hasMoreElements(); ) {
                            synonims.add(ias.nextElement().getHostAddress());
                        }
                    }
                }
                localhostIpSynonyms = synonims;
                logger.info("Local IP's resolved to:" + localhostIpSynonyms);
                return localhostIpSynonyms.contains(ip);
            } catch (Exception e) {
                logger.error("Can not resolve list of local ip adress", e);
            }
        }
        return false;
    }

Trick is not only to rezolve hostname to ip with port set, but to lookup all local interfaces in order to drop local ones.

Trunk Based Development vs Successful-git-branching-model

At work I've faced 2 generic approaches to branching:

1. "Modern", where we have branch-per-feature policy:
   http://nvie.com/posts/a-successful-git-branching-model/
2. "Classic", where everybody commit to the trunk and branches are made only for releases:
   http://paulhammant.com/2013/04/05/what-is-trunk-based-development/
   

Both approaches are very interesting, but my personal favorite is "Classic" one. Because:

1. CI  (Continious integration)- you merge often, and you have ci server testing your code. In first approach, you'll need job-per-branch, and there are a lot of them! In second, one job will notify you then your commit is breaking something
2. If your changes are not compiling\not all test pass? Well, in this case solution is to commit but don't push.
3. Second approach forces some sporadic code review, when 2 developers change some linked source - and this triggers communication and is a good way to common vision.