Commonly used Java libraries

Java is a classical general purposes language that can do most of the things. Java programming constructs are too primitive that people are complaining about writing redundant code, like sorting, searching, set union or intersection and etc on their own.

In fact, there are ready to use utility libraries that could save us hours in writing common logics, the only matter is the programmer attitude towards re-using others’ code or re-inventing the wheels.

Most of the utilities come from the following sources. I always search before I write code.
1. JDK
2. Apache Common
3. Spring Utilities
4. Google Guava

Here are my favourite lists of libraries.

JDK Utilities
JDK has comes with comprehensive functions for Collections(List, Set, Map) and Arrays. Most collection related functions like Sorting, Searching, Union, Swapping, Reverse orders are already supported, which means that writing 2 nested for-loop for sorting and one for loop for searching are too outdated.
java.util.Collections
https://docs.oracle.com/javase/7/docs/api/java/util/Collections.html
java.util.Objects
https://docs.oracle.com/javase/7/docs/api/java/util/Objects.html
java.util.Arrays
https://docs.oracle.com/javase/7/docs/api/java/util/Arrays.html

IO Related Utilities
When we deal with Java Streams, Reader, Writer and Files, writing buffer reading logics are too comsy and hard to get it right, like the try-catch-finally structure for Streams and Readers. IOUtils provides those static method for it.
org.apache.commons.io.FileUtils
https://commons.apache.org/proper/commons-io/javadocs/api-2.6/org/apache/commons/io/FileUtils.html
org.apache.commons.io.IOUtils
https://commons.apache.org/proper/commons-io/javadocs/api-2.6/org/apache/commons/io/IOUtils.html

String Utilities
There are String utilities comes from different libraries, they usually provide functions like substring, joining, regex matching, splitting, extraction and search & replace . Some of the functionality are overlapping, so, you need to look at the API before you start.
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/util/StringUtils.html
https://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/StringUtils.html
https://google.github.io/guava/releases/19.0/api/docs/com/google/common/base/Strings.html

Bean Utilities
The Java Reflection API is inherently hard to use and error prone, we need to handle properties accessor, access level, value getter and setters. Spring provides BeanUtils and BeanWrappers that make this kind of access much easier.
org.springframework.beans.BeanUtils
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/beans/BeanUtils.html
BeanWrapper and BeanWrapperImpl
We can create a BeanWrapper and access the property value by property name (String).
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/beans/BeanWrapper.html
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/beans/BeanWrapperImpl.html

More advanced data structure
Google Guava provides advanced data structure, like MultiSet (Count for an object occurrence), Multimap (Group objects under the same key), BiMap (Key->Value AND Value->Key mapping) and etc. You can think twice when you deal with Map or Map> which looks very complicated.
https://github.com/google/guava/wiki/NewCollectionTypesExplained

More collections utilities from Guava
Google Guava provides more advanced Collection related utilities, which is not provided by JDK Collections class, like Cartesian Products, Subset operation and etc. It may also be first steps to Java Functional programming and bridge to Java 8 Lambda expression.
https://github.com/google/guava/wiki/CollectionUtilitiesExplained

Self Reflection on IT company structure

Interesting stories to share

https://dev.to/henrylim96/reverse-engineering-cathay-pacifics-seat-selection-page-43od

I am imagining how’s CX handle this incident internally and how to avoid. CX “supposes” to have good system and control, every thing should have check and balance. Who should be responsible for this?????

Imagining there are standard in-house software development structure, different teams would have claims as followed.
Business User: IT is shit, making rubbish, charge me so much (transfer pricing). Fire them all!!! (Yes, they did, I think they deserve)

Business Analyst: I have already documented the user requirement and expectation, modifying the ticket class is not a valid use case, it should be security team responsibility, definitely not my fault.

Security Team: My responsibility is using the million dollars app scanner, network scanner, IDS (Intrusion detection system) and XYZXYZ (lots of buzz words) to do regular checking, I just know scanning, but nothing about business.

Dev Team: Such validation is not written on the specification, it makes no sense for me to implement it.

Micro-service Dev Team: This logic suppose to be validated by XXX Team, it is not my responsibility to re-validate and I am NOT TOLD TO DO SO.

Architect: (Playing fingers) It is business use case, not on my dish.

QA: BA, pls confirm(The requirement). DB Team, pls confirm. Dev Team, pls confirm. I don’t know who should I ask to confirm. I am just a test plan executor. I can be BA if I know the business well, I could be a programmer if I can code test case. This incident is definitely not my issue.

DB Team: I only deal with DB Structure and constraint.

Support / Customer Service: The phones are all ringing, the customer has fxxked us so hard. Dev Team, pls advice. BA, pls advice. DB Team, pls advice. Architect Team, pls advice.

Internal Audit Team: I am just a Business Man, knowing how to present and tender external party for auditing. I don’t really know how the system works, how could I audit to this level?

The management may claim everyone is responsible, but eventually it means no one is responsible.

It is ironic that simple script kiddie technique can break several million dollars project, and destroy the brand. I don’t think this is the only bug on the system or any other multi billions dollars organization, from banks, to hospitals, to varies online providers.

Disclaimer: Any similarity is mere coincidence

Setup IBM MQ on Ubuntu

My recently is recently integrating with a banking solution via IBM MQ, I need to install an IBM MQ for testing. Here are my steps for the installation

PS. IBM MQ 9.1 requires IPv6 for all communication, so, I fall back to IBM MQ 8.0

1. Download IBM MQ binary from IBM Website
https://developer.ibm.com/messaging/mq-downloads/

2. Follow the following link for the installation
https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.ins.doc/q008640_.htm

3. After installation, add ubuntu user (current user) to the “mqm” group

sudo usermod -aG mqm $USER

4. Up to now, the installation should be completed.

5. Setup Env and Queue Manager

There is a default MQ Installation (Installation1) setup, you can use the installation directly.

The program directory is /opt/mqm , you can find all command in /opt/mqm/bin , all subsequent execution are in that folder.

6. Create the MQ Manager

crtmqm QM1
strmqm QM1

7. Download the sample script from the following link, it will create some default queue and settings.

https://raw.githubusercontent.com/ibm-messaging/mq-dev-samples/master/gettingStarted/mqsc/mq-dev-config.mqsc

Modify the script, change the user from “app” to “ubuntu”

DEFINE CHANNEL(‘DEV.APP.SVRCONN’) CHLTYPE(SVRCONN) MCAUSER(‘ubuntu’) REPLACE

Execute the script

./runmqsc QM1 < ~/mq-dev-config.mqsc

8. MQ Server uses Linux user and user group for authentication and authorization, so, we need to assign more MQ Server privilege to Linux user group (mqm)

./setmqaut -m QM1 -t qmgr -g mqm +connect +inq
./setmqaut -m QM1 -n DEV.** -t queue -g mqm +put +get +browse +inq

9. Finally, since IBM MQ 7.5, Channel would also require authentication, but I have no idea on how it works yet, so the simplest solution is disabled that.

./runmqsc QM1
ALTER QMGR CHLAUTH(DISABLED)

PS. We should figure out how to work properly with the channel authentication.

http://www-01.ibm.com/support/docview.wss?uid=swg21577137

10. Testing with MQ Explorer

Install IBM MQ Explorer from the following link

https://www-01.ibm.com/support/docview.wss?uid=swg24021041

You can use MQ Explorer to connect to .39 MQ Queue Manager.

Build my Fusion Drive with LVM-Cache

LVM Cache is a interesting feature that I saw some web review on the web. It can speed up traditional mechanical disk with a cache partition on SSD. Essentially the concept is the same as Windows Readyboost or Fusion Drive, which the caching is controlled by firmware.

Resize the SSD

e2fsck /dev/sda1

resize2fs /dev/sda1 100000M

Use GParted to adjust the partition size

Mark Data(sdb1), Cache(sda2) and Meta(sda3) as LVM PV

sudo pvcreate /dev/sdb1 

sudo pvcreate /dev/sda2

sudo pvcreate /dev/sda3

Create Volume Group with the PVs

# Must spare some space
sudo vgcreate VG /dev/sdb1 /dev/sda2 /dev/sda3

sudo lvcreate -l 95%FREE -n data VG /dev/sdb1

sudo lvcreate -l 95%FREE -n cache VG /dev/sda2

sudo lvcreate -l 95%FREE -n meta VG /dev/sda3

Create Cache Pool and config cache mode as Writeback (improve read / write performance)

sudo lvconvert --type cache-pool --poolmetadata VG/meta VG/cache

sudo lvconvert --type cache --cachepool VG/cache --cachemode writeback VG/data

In case you are unlucky that u encounter the cache corruption. You need to execute the following commands to rebuild the cache

vgchange -a y VG
lvchange -a y VG/data
lvconvert --repair VG/data

Reference:
http://man7.org/linux/man-pages/man7/lvmcache.7.html

ActiveMQ Artemis quick start notes

I have played around with Artemis for a few days, here are the useful command I used. The exact config is stored at my github.

https://github.com/jimmysyss/artemis-config

1. Create 2 brokers with port offset
artemis create mybroker1 –port-offset 0
artemis create mybroker2 –port-offset 0

2. Testing the producer and consumer
./artemis producer –url tcp://localhost:61616
./artemis consumer –url tcp://localhost:61616

3. Enabled clustered
See jgroup.xml and broker.xml , and

4. Load balancing
See broker.xml, Add in

5. Enable SSL for HornetQ OpenWire protocol
a. Provide keystore.jks and truststore.jks
b. Config the acceptor in broker.xml

Nginx per user directory

It is common for hosting company to share the same host and create per user directory, so that every user can browse with the following url.


http://localhost/~jimmy/index.html

It can be achieved easily by nginx configuration.


sudo vi /etc/nginx/sites-available/default

Add the following code under “server” section

location ~ ^/~(.+?)(/.*)?$ {
alias /home/$1/www$2;
autoindex on;
}

Add the user to www-data group

sudo usermod -aG www-data $USER

Change the www directory under user home to mod 755, so that others can access the folder with execute right

chown 755 -R ~/www/

MiniDLNA configuration

MiniDLNA is a uPnP server that allow you to play Video, Picture and Music over LAN via IP broadcasting protocol.

I have installed it in my Raspberry Pi as a mini-server at home, to stream video to my TV and my XiaoMi TVBox.

By default, it runs with “minidlna” user, it may not be so convenient for use because it is not a usual user, I need to manually copy files to the Pi and subsequently to the folder accessible by minidlna. It is related to the user privilege issue. Funny enough, simply change on folder to 666 doesn’t solve the problem because minidlna requires to access the folder along the path from “/”

Therefore, the only solution is to configure MiniDLNA to run on another user account rather than the default minidlna user.

The minidlna.conf is as followed


# Specify the user name or uid to run as.
user=pi
....
#media_dir=/var/lib/minidlna
media_dir=/home/pi/DLNA
....
#db_dir=/var/cache/minidlna
db_dir=/home/pi/.dlnacache
....
#log_dir=/var/log
log_dir=/home/pi/.dlnalog

Some notes

sshpass -p ‘*******’ rsync -avzh -e ‘ssh -p 40022’ XXXXFILEXXXX pi@XXXXX:Downloads/

# Search for my current IP
dig +short myip.opendns.com @resolver1.opendns.com

We can use openssl command to verify the connection.

openssl s_client -connect 75.124.180.124:11405 -cert cont-public.cert -key cont.key

In order to create a java keystore, we have to create a P12 keystore first and then use keytool to import as Java Keystore. The commands are as followed.

openssl pkcs12 -export -in cont-public.cert -inkey cont.key -out cont20161209.p12 -name initiator
keytool -importkeystore -destkeystore initiator.jks -srckeystore cont20161209.p12 -srcstoretype PKCS12 -alias initiator

Finally, we “Trust” the public certificates

keytool -import -trustcacerts -file ullink.cert -alias ullink -keystore initiator.jks

And we can verify with the following command

keytool -list -v -keystore initiator.jks

What is Software Architecture?

Software architect looks like a prestige job, but to me, it is nothing different from an engineering with a deeper understanding about the problem domain and limitations.

As an architect, I look at my job from different dimensions. This ability is very important or you will fall into the trap of over-engineering or under design(It may be the case for my daily life).

1. Business Dimension: Understand the problem

The key difference between a software architect and a general programmer is not simply the technical competence, it should be the degree of understanding to a business domain. Without such understanding, an architect cannot make reasonable trade-off among different solutions. It covers the flexibility, configuration, performance and loading concern.

2. Structural Dimension: How the code are organized and grouped?

The main difference between a good program and bad program is how the program is structured so that it can truly reflect the current and upcoming business use cases. I suggest to start small, and try to explain the function of each module in “1 simple sentence”. You can refactor the project later on if your business grows.

3. Tier Dimension: Classical MVC? Restful => Spring => ESB => DB?

Once we defined a business use cases, we can design the data flow. Usually data or request will flow through different layers which have different concerns. MVC is a classical 3-tier architecture, which C & V are external facing while M are the business objects, we can then focus on the responsibilities of each tier. Furthermore, there are more layering architecture, for example, Events => Karfka => Cassandra for data warehouse. We need to broaden our eye to understand how others solve similar problems

4. Library Dimension: What and why we can choose

We should not reinvent the wheel, when we want to implement something, we should first Google to see if there is any available libraries which serves the purposes. Using other’s libraries can save your time and most importantly, keep some design frauds away from your system. Of course, if you are an experienced architect, you can quickly sniff a library fit your needs or not, it is the key value you added to your team.

Install Superset with Ubuntu 16.04 with venv


sudo apt-get install build-essential libssl-dev libffi-dev python-dev python-pip libsasl2-dev libldap2-dev python3-dev

python3 -m venv superset-venv

source superset-venv/bin/activate

pip install --upgrade setuptools pip

pip install superset

# The following copy from https://superset.incubator.apache.org/installation.html

# Create an admin user (you will be prompted to set username, first and last name before setting a password)
fabmanager create-admin --app superset

# Initialize the database
superset db upgrade

# Load some data to play with
superset load_examples

# Create default roles and permissions
superset init

# Start the web server on port 8088, use -p to bind to another port
superset runserver