Showing posts with label java. Show all posts
Showing posts with label java. Show all posts

May 1, 2017

Cost of regular expressions in java

Recently I've came across an interesting case of names transformation to so called "slug" format:
Step 1: covert to lowercase
Step 2: replace any alpheNumeric to hypen(-). Start to End of the string: Don't consider unicodes
Step 3: Remove any adjacent hypen(-)
Step 4: Remove trailing or leading hyphens
 First, quite readable solution:

    public static final String CHARS_REG = "[^a-zA-Z0-9]";
    public static final String DOUBLE_REG = "[-]+";
    public static final String TRIM_REG = "-$|^-";
    public static final String DASH = "-";
    static final Pattern CHARS = Pattern.compile(CHARS_REG);
    static final Pattern DOUBLE = Pattern.compile(DOUBLE_REG);
    static final Pattern TRIM = Pattern.compile(TRIM_REG);

    public static String makeSlugNamePatterny(String name) {
        String it = name.toLowerCase().trim();
        it = CHARS.matcher(it).replaceAll(DASH);
        it = DOUBLE.matcher(it).replaceAll(DASH);
        it = TRIM.matcher(it).replaceAll("");
        return it;
    }
Unfortunately, out-of-the-box java regexp support can not trim and lowercase, so we see 5 operations here. It was curious for me to implement it regex-free and to compare performance:
     public static String makeSlugNameOptimized(String name) {
        String result = name + " ";
        StringBuilder temp = new StringBuilder();
        char ch1 = 0;
        char ch2 = toLowCaseAlfanumeric(result.charAt(0));
        for (int i = 0; i < result.length() - 1; i++) {
            ch1 = ch2;
            ch2 = toLowCaseAlfanumeric(result.charAt(i + 1));
            if (ch2 != ch1) {
                temp.append(ch1);
            } else if ('-' != ch1) {
                temp.append(ch1);
            }
        }

        while ('-' == temp.charAt(0)) {
            temp.deleteCharAt(0);
        }
        while ('-' == temp.charAt(temp.length() - 1)) {
            temp.deleteCharAt(temp.length() - 1);
        }

        return temp.toString();
    }

    private static final int Aa = 'a' - 'A';

    private static char toLowCaseAlfanumeric(char c) {
        if (c >= 'a' && c <= 'z') {
            return c;
        }
        if (c >= 'A' && c <= 'Z') {
            return (char) (c + Aa);
        }
        if (c >= '0' && c <= '9') {
            return c;
        }
        return '-';
    }
Note that optimized case doesn't trim or lowercase separately - it's a part of  the loop.
On 500 random strings 500 times gave following performance: patterny-2059 milliseconds, optimised - 277.

Conclusion

10x time improvements  - it's the cost of regexps in my case. Readability can be improved by extracting some methods, so it should not be a concern here. Generally speaking, either regexp is simple - in this case you can work around like I did, or it's complex - in this case it's both slow and unreadable.  
Basically,  as classic(https://xkcd.com/1171/) says: 

May 27, 2016

Pass arguments to your main method in gradle bootRun

If you are tired of Spring Boot configuration magic, there is one more trick to confuse you completely.
How to configure Spring to work without knowing profile or environment? Dropwizard style, like this:
java -jar myjar.jar myconfig.yml.
No profiles, no wandering how did my properties got populated.

Spring Boot first class configuration is Java bean. Unfortunately, due to legacy issues, we need to include xml config. After some pain, suffering and reading, I found solution:
@ImportResource("classpath:application-context/applicationContext.xml")
//@Component
public class BatchConfiguration 

In similar way you can include properties file.
Next step is ${properties}. In xml configuration there are some environment-specific properties, like database url and so on. Some advise JNDI, but it introduces one more layer of magic - configuration of a web container.
So, question is, how to populate properties with external config. There are a ton of solutions out there, but none worked for me. Maybe because it's Spring Boot+Spring Data+Spring Batch or maybe because I don't understand this page. Anyway, I found my own way:
    
public static void main(String[] args) throws IOException {
        if (args.length >= 1) {
            Properties p = new Properties();
            p.load(new FileReader(args[0]));
            p.forEach((x, y) -> {
                System.setProperty((String) x, (String) y);
            });
        }
        SpringApplication.run(Application.class, args);
    }
Now it works perfectly in fatJar task, but what about bootRun? Boot Run saves a lot of dev time and is very easy to use. As you probably know, BootRun extends gradle standard JavaExec task, with awesome parameter args. It goes straight to your main method. So gradle bootRun task looks like:
bootRun {
    args =["cars-etl.properties"]
    
}
This is all for make your solution work! Have fun.
P.S. HowTo pass jvmArgs
P.P.S. HowTo run job from Controller

May 25, 2016

MongoBulkItemWriter

Spring Batch is a nice choice for simple ETL jobs, but it doesn't work well with mongodb, especially writing to it. Provided in Spring Batch MongoItemWriter doesn't do bulk inserts.
Fortunately for us, bulk inserts are quite easy to implement:
import com.mongodb.BulkWriteOperation;
import com.mongodb.BulkWriteResult;
import com.mongodb.DBObject;
import org.springframework.batch.item.ItemWriter;
import org.springframework.data.mongodb.core.MongoTemplate;

import java.util.List;

public class MongoBulkItemWriter<T> implements ItemWriter<T> {

    private String collection;
    private MongoTemplate template;

    public MongoBulkItemWriter(String collection, MongoTemplate mongoTemplate) {
        this.collection = collection;
        this.template = mongoTemplate;
    }

    @Override
    public void write(List items) throws Exception {
        BulkWriteOperation bulk = template.getCollection(collection).initializeUnorderedBulkOperation();
        items.forEach(i->{
                bulk.insert((DBObject) template.getConverter().convertToMongoType(i));
        });
        BulkWriteResult result = bulk.execute();
    }
}
It works much faster, but beware - inserts only, so item with duplicate id will ruin your batch. Solution might look something like this:
        BulkWriteOperation bulk = template.getCollection(COLLECTION_NAME).initializeUnorderedBulkOperation();
        updates.forEach(u -> {
            bulk.find(new BasicDBObject("id", u.getId())).upsert().update(u.getDbObject());
        });
        bulk.execute();
Upserts are much slower than pure inserts, but still a huge win compared with per object writes.

Jan 15, 2016

OOP - we are fooling ourselves

Object Oriented Programming is a holy grail of modern code-writing methodologies. As everything so fundamental, it was criticised  quite heavily, with different degree of details. Some say, OOP is dead, some say it’s harmful. I think, we should reconsider our understanding of this paradigm.
If we talk about objects - we think about state (a lot) and behaviour (a little) . But in all domain models  we separate between «domain» and «business» logic. 

Example: A human buys bananas in a grocery store. Here set of objects Bananas change ownership and location. Object Store obtains money, frees some space in a shop, and loose some bananas. Object human looses money and gets a bananas. 
Question: What object should own this logic? If it’s 3 different objects, each with it’s own functions, then who should coordinate behaviour? Is it a shop who should take money from a human and change ownership of bananas? Or Human? Or Banana? Or some different different Object?
In practice, usually it’s different object, most probably StoreService. 

After that we have another question - about persistence. Somebody should store it to a database (persistent storage). Should it be domain objects? In practice, it’s usually DAO layer. 
And I will not dig into details - because human usually need to pack bananas in a bag and store should manage not to run out of bananas.
So, what do we have here - our objects are passive data objects, and all behaviour is in other objects, with some dependency on external resources (database).

Is it OOP ? Well, in practice, it’s what everybody calls OOP. Clearly, something is wrong.
Looking on a Store, we will not have an idea about all it’s abilities. StoreService will give us much more information. 

There is one solution I know to this problem - mix-ins. You will be able to mix some resource-dependent behaviour into domain objects, without altering domain objects much. Is it OOP ?
Nice articles on a subject:

Nov 11, 2013

Hazelcast trick 1

I was playing with hazelcast for a while. Great tool for java-based distributed systems - simple and elegant.
During our work I've run into some tricky situations. I understand that they can be solved in different ways, I've shosen to write some bits of code. I'll share these bits with explanations in upcoming posts.

Hazelcast Trick 1. Multicast vs TcpIpJoiner

Problem: 

In our network some clusters were in several subnets - and I've not figured out how to fix this yet, but hazelcast has tcp-ip joiner to the resque. There is one disadwantage though. Same config file for different machines makes it hard not to include localhost in config. And hazelcast was not working with hostnames correctly for me.

Solution:

So, I've decided to keep config file uniform, but add following code to my startup procedure:
       Config cfg = new FileSystemXmlConfig(configFile);
        cfg.setInstanceName(configName);
        if (tcpipjoin != null && tcpipjoin.isEnabled()) {
            cfg.getNetworkConfig().getJoin().getTcpIpConfig().clear();
            cfg.getNetworkConfig().getJoin().getTcpIpConfig().setEnabled(true);
            cfg.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
            for (String host : tcpipjoin.getHosts()) {
                String resolvedHost = resolveHostToIp(host);
                if (resolvedHost != null) {
                    cfg.getNetworkConfig().getJoin().getTcpIpConfig().addMember(resolvedHost);
                }
            }
        }
This solved 2 issues:

  1. Resolve hostname to IP (oddly, hazelcast was not doing well with hostnames, dont know why yet)
  2. Exclude localhost from list - in this final configuration was simple and effective. Without attempting to call itself and rezolve hostname.
Helper methos is not as straitforward, here it goes:

    static String resolveHostToIp(String host) throws UnknownHostException {
        String[] parts = host.split(":");
        if (parts.length > 2) throw new UnknownHostException("Illegal host format: " + host);
        String ip = InetAddress.getByName(parts[0].trim()).getHostAddress();
        if(isLocal(ip)) {
            logger.info("Host " + host + "filtered out as local");
            return null;
        }
        String resolved = parts.length == 1 ? ip : (ip + ":" + parts[1].trim());
        logger.debug("host " + host + " resolved to " + resolved);
        return resolved;
    }

    private volatile static List localhostIpSynonyms;

    private static boolean isLocal(String ip) {
        if (localhostIpSynonyms != null) {
            return localhostIpSynonyms.contains(ip);
        }
        else {
            try {
                List synonims = new ArrayList<>();
                for (Enumeration ifcs = NetworkInterface.getNetworkInterfaces(); ifcs.hasMoreElements(); ) {
                    NetworkInterface ifc = ifcs.nextElement();
                    if (ifc.isUp()) {
                        for (Enumeration ias = ifc.getInetAddresses(); ias.hasMoreElements(); ) {
                            synonims.add(ias.nextElement().getHostAddress());
                        }
                    }
                }
                localhostIpSynonyms = synonims;
                logger.info("Local IP's resolved to:" + localhostIpSynonyms);
                return localhostIpSynonyms.contains(ip);
            } catch (Exception e) {
                logger.error("Can not resolve list of local ip adress", e);
            }
        }
        return false;
    }

Trick is not only to rezolve hostname to ip with port set, but to lookup all local interfaces in order to drop local ones.

Oct 12, 2012

Schedule at fixed time

Extract of some useful tip I've posted on stackoverflow,
how to schedule task at fixed time at specified timezone:

 
timer = new Timer("Timer", true);
    Calendar cr = Calendar.getInstance(TimeZone.getTimeZone("GMT"));
    cr.setTimeInMillis(System.currentTimeMillis());
    long day = TimeUnit.DAYS.toMillis(1);
    //Pay attention - Calendar.HOUR_OF_DAY for 24h day model 
    //(Calendar.HOUR is 12h model, with p.m. a.m. )
    cr.set(Calendar.HOUR_OF_DAY, it.getHours());
    cr.set(Calendar.MINUTE, it.getMinutes());
    long delay = cr.getTimeInMillis() - System.currentTimeMillis();
    //insurance for case then time of task is before time of schedule
    long adjustedDelay = (delay > 0 ? delay : day + delay);
    timer.scheduleAtFixedRate(new StartReportTimerTask(it), adjustedDelay, day);
    //you can use this schedule instead is sure your time is after current time
    //timer.scheduleAtFixedRate(new StartReportTimerTask(it), cr.getTime(), day);
 

Oct 1, 2007

How to write javadoc

My personal opinion how to write good javadocs (with examples from jdk)

Overview of method/class

Goal of method/class ? What is goal of class. Example:
* The <code>String</code> class represents character strings. All
* string literals in Java programs, such as <code>"abc"</code>, are
* implemented as instances of this class.

Responsibility of method/class ?What method/class is doing. Example in code above.

Action of method ?What statements we can make about how goal is achieved ? Example in StringBuffer class.
/**
* Ensures that the capacity of the buffer is at least equal to the
* specified minimum.
* If the current capacity of this string buffer is less than the
* argument, then a new internal buffer is allocated with greater
* capacity. The new capacity is the larger of:
* <ul>
* <li>The <code>minimumCapacity</code> argument.
* <li>Twice the old capacity, plus <code>2</code>.
* </ul>
* If the <code>minimumCapacity</code> argument is nonpositive, this
* method takes no action and simply returns.
*
* @param minimumCapacity the minimum desired capacity.
*/
public synchronized void ensureCapacity(int minimumCapacity) {

And, of course, describe boundary situations carefully.

Preconditions of method:

Some special agreement then it's legal to apply method/class, example:
* @param s a <code>String</code> containing the <code>int</code>
* representation to be parsed

It's not necessary must be placed in params, but it's good to have such statement. ;)

Postconditions of method:

What it must return if everything is ok.
* @return the integer value represented by the argument in decimal.

Exceptions documentation:

Please don't provide javadoc like "throws Exception if something goes wrong", it has no information in it.
Good Example is:
* @exception NumberFormatException if the string does not contain a
* parsable integer.

CRC description of class

Class javadoc is good then it contains Overview of class, it's responsibilities, classes it coolaborates with, and brief description of working logic and purpose. Example from Timer class.
/**
* A facility for threads to schedule tasks for future execution in a
* background thread. Tasks may be scheduled for one-time execution, or for
* repeated execution at regular intervals.

Examples of use helps a lot sometimes.

Links:

java.sun.com/j2se/javadoc/writingdoccomments/

alistair.cockburn.us/index.php/Structuring_use_cases_with_goals

en.wikipedia.org/wiki/Design_by_contract