Part 2. Why software outsourcing companies should be worried that business is booming

Continuation of my previous post
This blog post will speculate on Innovation and Automation processes and how they might affect the IT outsourcing business.

Fictional, yet quite probable story of one project in one bank

 In a galaxy far, far away, one bank decided to deprecate its legacy system and re-create a project, in order to increase its flexibility and deliver more features that will reliably provide
a greater amount of revenue.

Chapter 1. Team & Project

How we should reassign team & project in a safe way. There are several obvious choices

1. Same team, in-house - probably, new system will be just old system rewritten - same technology, same architecture.  It might give some benefits, but overall delivery & time will suffer. Additional hire is possible, with question - what to do
   with extra people once migration is be over?
2. 2 teams, in-house - take most promising people from old project, hire some new ones and re-create project with a new approach.
3. 2 teams, in-house old project and outsource new one -  promote someone from old team, so he will be transferring  his knowledge to some smart people offshore.
4. Outsource support + develop new project in-house (or mixed) - leave undesired people coaching how how to fix bugs in the old system, take  brilliant in-house engineers and mix them with some talented contractors 

Chapter 2. Automation

 Every project needs infrastructure. What will be your choice ?

1. Have same old reliable manual deployment process - +20% to development time, +1 Ops guy
2. Have separate automation engineer - -10% development time, +1 devOps
3. Have everyone ( or at least 2ppl) in a new project team who can support automated deployment solution (CD) process - -10% dev time, requires up-front investment, +1 devOps

Would you outsource automation of CD process ? What is something goes wrong ?

Chapter 3. Innovation

What technology and approach to use for the new project ?

1. Same technology, same approach  - who needs innovation anyway ?
2. New technology - learning curve might be steep, risks of instability are high (really?), potential profit - fast time to market with less maintenance.

Here I want to add some specifics. Let's compare Java and Scala :

1.  Stability - Scala was presented in 2003, so it's pretty stable now. Many fall victims of the familiarity bias.
2.  Quality - less code means less bugs
3.  Productivity - once you master it (it's more complex than java), you can be much faster than before (survey result with numbers)

Chapter 4. Conclusions and alternative endings

Now let's say there are 100 companies on the same market. How many of them can survive with ch1.3 approach (outsource new stuff)? It is much riskier than ch1.4 and in 2 years perspective quite suicidal  for the business (As in Funky Business - outsource all but critical).
Let's say 30 of them will go with ch1.4 approach.  10 will choose ch2.3 and 5 will go with ch3.2. 3 of them will go bankrupt due to various reasons, new technology included.
So 2 out of 100 projects will have time to market at least twice as short as market average.  Management in these projects will listen to their developers, and developers will listen to product owners. I would definitely buy stocks of these companies.
In 4 years perspective, what companies/projects will survive and flourish ? I think the ones
that have chosen ch1.4+ch2.3+ch3.2 Do you think these companies will be outsourcing new projects/new technologies ? Quite possible. Will service providers be ready for it ?

Part 1. Why software outsourcing companies should be worried that business is booming

I'll show why booming business might not be such a good thing for IT outsourcing companies.
The trend looks good for outsourcing providers, at least for INFOSYS, EPAM, LUXOFT. Quarterly (yoy) growth 13.30% -32.20%, with steady positive cash flow. But can it be the beginning of the end?

5 bullet points

IT outsourcing growth might be linked with:

• Merge and Acquisition  - since the market growth is negative, big companies thrive , while small ones go up in smoke.
• Product companies outsource deprecated technologies  - in-sourcing new projects with new technologies, because innovation matters. "And it is often difficult to get innovative, differentiated outcomes when you turn IT over to someone else".

This leads to the following risks:

• Knowledge deprecation - outsourcing providers are not early adopters or innovators for obvious reasons, stated in the excellent "The Passionate Programmer" book.
• In-sourcing of IT jobs - there's a movement to bring IT jobs back to USA and look for local talent. It's relatively small, but has quite important objectives - to reduce risk, and lower management and communication overheads.
• High automation of programmer's work - with sharp up-to-date skills and tons of open source tools, one passionate programmer can replace 10 average ones or 20 outsourced (factor in communication overhead [ 2.1 The Formula chapter in the link ]).

Technology adoption challenge

In a future blog post I'll speculate about the Innovation and Automation points above, in particular how emerging languages can affect IT business.

Summary

The outsourcing boom might be linked with unimportant projects with deprecated technologies at the end of their life-cycle, and can can turn into durst pretty quickly.

Links

Deloitte’s 2014  Global Outsourcing  and Insourcing Survey  
10 IT Outsourcing Trends to Watch in 2014

Pass arguments to your main method in gradle bootRun

If you are tired of Spring Boot configuration magic, there is one more trick to confuse you completely.
How to configure Spring to work without knowing profile or environment? Dropwizard style, like this:
java -jar myjar.jar myconfig.yml.
No profiles, no wandering how did my properties got populated.

Spring Boot first class configuration is Java bean. Unfortunately, due to legacy issues, we need to include xml config. After some pain, suffering and reading, I found solution:

@ImportResource("classpath:application-context/applicationContext.xml")
//@Component
public class BatchConfiguration 

In similar way you can include properties file.
Next step is ${properties}. In xml configuration there are some environment-specific properties, like database url and so on. Some advise JNDI, but it introduces one more layer of magic - configuration of a web container.
So, question is, how to populate properties with external config. There are a ton of solutions out there, but none worked for me. Maybe because it's Spring Boot+Spring Data+Spring Batch or maybe because I don't understand this page. Anyway, I found my own way:

    
public static void main(String[] args) throws IOException {
        if (args.length >= 1) {
            Properties p = new Properties();
            p.load(new FileReader(args[0]));
            p.forEach((x, y) -> {
                System.setProperty((String) x, (String) y);
            });
        }
        SpringApplication.run(Application.class, args);
    }

Now it works perfectly in fatJar task, but what about bootRun? Boot Run saves a lot of dev time and is very easy to use. As you probably know, BootRun extends gradle standard JavaExec task, with awesome parameter args. It goes straight to your main method. So gradle bootRun task looks like:

bootRun {
    args =["cars-etl.properties"]
    
}

This is all for make your solution work! Have fun.
P.S. HowTo pass jvmArgs
P.P.S. HowTo run job from Controller

MongoBulkItemWriter

Spring Batch is a nice choice for simple ETL jobs, but it doesn't work well with mongodb, especially writing to it. Provided in Spring Batch MongoItemWriter doesn't do bulk inserts.
Fortunately for us, bulk inserts are quite easy to implement:

import com.mongodb.BulkWriteOperation;
import com.mongodb.BulkWriteResult;
import com.mongodb.DBObject;
import org.springframework.batch.item.ItemWriter;
import org.springframework.data.mongodb.core.MongoTemplate;

import java.util.List;

public class MongoBulkItemWriter<T> implements ItemWriter<T> {

    private String collection;
    private MongoTemplate template;

    public MongoBulkItemWriter(String collection, MongoTemplate mongoTemplate) {
        this.collection = collection;
        this.template = mongoTemplate;
    }

    @Override
    public void write(List items) throws Exception {
        BulkWriteOperation bulk = template.getCollection(collection).initializeUnorderedBulkOperation();
        items.forEach(i->{
                bulk.insert((DBObject) template.getConverter().convertToMongoType(i));
        });
        BulkWriteResult result = bulk.execute();
    }
}

It works much faster, but beware - inserts only, so item with duplicate id will ruin your batch. Solution might look something like this:

        BulkWriteOperation bulk = template.getCollection(COLLECTION_NAME).initializeUnorderedBulkOperation();
        updates.forEach(u -> {
            bulk.find(new BasicDBObject("id", u.getId())).upsert().update(u.getDbObject());
        });
        bulk.execute();

Upserts are much slower than pure inserts, but still a huge win compared with per object writes.

Spring Data Mongo Criteria Example

spring-data-mongodb-1.8.4 has a great package , with some features that makes mongodb queries easy. Unfortunately, documentation on this topic is not very good. I'll contribute to it by short example.

Problem

We want to search mongodb using $and and $in keyword based on several fields. But we don't know field configuration beforehead. Input params are fit into domain class:

public class MakeModelYear {
    private String[] make;
    private String[] model;
    private String[] year;

    public MakeModelYear(String[] make, String[] model, String[] year) {
        this.make = make;
        this.model = model;
        this.year = year;
    }

    public String[] getMake() {
        return make;
    }

    public String[] getModel() {
        return model;
    }

    public String[] getYear() {
        return year;
    }

    public boolean isEmpty() {
        return isEmpty(model) && isEmpty(make) && isEmpty(year);
    }

    public static boolean isEmpty(String[] a) {
        return a == null || a.length ==0;
    }
}

Any of make model and year can be null, and we still need to have nice smooth functionality

Solution

For Criteria object:

   public static Criteria generateQuery(MakeModelYear key) {
        Queue cril = new ArrayDeque<>();
        if (!MakeModelYear.isEmpty(key.getMake())) {
            Criteria makeCri = Criteria.where("Make").in(Arrays.asList(key.getMake()));
            cril.add(makeCri);
        }
        if (!MakeModelYear.isEmpty(key.getModel())) {
            Criteria modelCri = Criteria.where("Model").in(Arrays.asList(key.getModel()));
            cril.add(modelCri);
        }
        if (!MakeModelYear.isEmpty(key.getYear())) {
            Criteria yearCri = Criteria.where("Year").in(Arrays.asList(key.getYear()));
            cril.add(yearCri);
        }

        return cril.size() == 1 ? cril.poll() : cril.poll().andOperator(cril.toArray(new Criteria[]{}));
    }

For Query itself object:

        MakeModelYear key = new MakeModelYear(make, model, year);
        Query query = new Query();
        if(!key.isEmpty())query.addCriteria(DaoHelper.generateQuery(key));
        query.fields().exclude("_id");
        return mongoTemplate.getCollection(CARS).find(query.getQueryObject(), query.getFieldsObject()).toArray();

As you can see, we can specify both Criteria and configuration what to return and what to supress in one readable object.
I could not find better way in Jongo and mongo driver.

Hexagonal architecture and ESB (Enterprise service bus). Part 1.

This blog post will start comparing 2 architectural approaches. 

You probably have heard about ESB - architectural model to connect everything with everything. It's interesting for me to compare it to less known but quite interesting concept of Hexagonal architecture.
In enterprise there are usually a tons of interconnected services\applications, so let's evaluate these approaches from enterprise view.

Overview 

In a one sentence:

Hexagonal architecture(HA) - ‘’every’’ piece of functionality the application(service) offers is available through an API (application programmed interface) or function call.

Enterprise service bus(ESB) - mediator service that facilitates work of loosely coupled applications(services) that are expected to be independently deployed, running, heterogeneous, and disparate within a network.
 Nobody reads middle section, so I will leave it for next posts and jump directly to a conclusion.

Conclusion

Hexagonal architecture might seem less mature, but, compared to enterprise service bus, it provides a lot of practical benefits for multiple services integration. They might look uglier on a paper, but, in practice, they provide faster time-to-market software with better quality and simpler error handling.
Good news is, that HA is compatible with ESB, so I think best way is to use HA as a basic approach, adding some ESB as you need it.
In one sentence - design your application around your business domain, not integration solution.

Dropwizard vs Spring Boot

In Java world there are 2 most wide spread solutions for JSON REST technology. In this post I’ll cover some aspects of them, that seems of most importance to me. 

Dependency Injection

With Spring, you are limited to Spring DI - quite complex and hard to debug. With Dropwizard, you can choose any DI you like. I was using Guice. But beware of using javax.inject annotations. Dropwizard v0.9.2. uses fresh release Jetty, that was build using HK2 DI library. So, using  general annotations with some other DI solution may result in strange behavior. For Guice-HK2 there are some bridge solutions, but using DI library specific annotations will save you from trouble.

Rest Api

I personally find JAX-RS specification not very convenient, maybe because I prefer single routes file and dislike annotating classes with @Path. Spring annotations for REST API are slightly better in this regard.

Configuration

Dropwizard offers once convenient way to configure everything - yml. With Spring you have as much flexibility as you can handle, and it makes server hard to maintain. I guess Dropwizard configuration is more in line with java values - there is single way to do it, simple, clear,maintainable. In case of issue you always know where to look.

Deployment

Dropwizard oriented on jar-deployment, standalone jetty. Spring can do both jar and war (for tomcat). It is arguable, but I consider multiple deployment options as a plus - it doesn’t complicate maintenance much, but can simplify support in case of running multiple applications.

Static Resources & Templates

Dropwizard offers quite simple way to mix pages and JSON API. If we will use angular on client side - it result in quite elegant solution with no tweaks. With Spring Boot I've struggled to configure my application to work with HTML and JSON, with support of some static ( e.g. css) resources.

SummaryTable

This is my marks for different aspects of platform. Max mark is 5. The bigger-the better.

Criteria	Dropwizard	Spring Boot
DI	5	3
REST API	3	4
Configuration	5	3
Deployment	4	5
Static Resources & Templates	5	3
Total	22	18

Summary

Dropwizard is better IMHO, and it plays nice with most Spring projects, like Spring Security and Spring JDBC. Spring Boot probably more performant, but I have no data on this topic, and all performance depends mostly from performance of your database. Solutions are quite similar, so winner detected.

There are lots of other articles on topic, that cover slightly different areas than this one.

See http://blog.takipi.com/java-bootstrap-dropwizard-vs-spring-boot/  and http://www.schibsted.pl/2015/07/spring-boot-and-dropwizard-in-microservices-development/ for more.

OOP - we are fooling ourselves

Object Oriented Programming is a holy grail of modern code-writing methodologies. As everything so fundamental, it was criticised  quite heavily, with different degree of details. Some say, OOP is dead, some say it’s harmful. I think, we should reconsider our understanding of this paradigm.

If we talk about objects - we think about state (a lot) and behaviour (a little) . But in all domain models  we separate between «domain» and «business» logic. 

Example: A human buys bananas in a grocery store. Here set of objects Bananas change ownership and location. Object Store obtains money, frees some space in a shop, and loose some bananas. Object human looses money and gets a bananas. 

Question: What object should own this logic? If it’s 3 different objects, each with it’s own functions, then who should coordinate behaviour? Is it a shop who should take money from a human and change ownership of bananas? Or Human? Or Banana? Or some different different Object?

In practice, usually it’s different object, most probably StoreService. 

After that we have another question - about persistence. Somebody should store it to a database (persistent storage). Should it be domain objects? In practice, it’s usually DAO layer. 

And I will not dig into details - because human usually need to pack bananas in a bag and store should manage not to run out of bananas.

So, what do we have here - our objects are passive data objects, and all behaviour is in other objects, with some dependency on external resources (database).

Is it OOP ? Well, in practice, it’s what everybody calls OOP. Clearly, something is wrong.

Looking on a Store, we will not have an idea about all it’s abilities. StoreService will give us much more information. 

There is one solution I know to this problem - mix-ins. You will be able to mix some resource-dependent behaviour into domain objects, without altering domain objects much. Is it OOP ?

Nice articles on a subject:

http://loup-vaillant.fr/articles/deaths-of-oop

http://www.smashcompany.com/technology/object-oriented-programming-is-an-expensive-disaster-which-must-end