Author Archives: jimmy

PostgreSQL + TimescaleDB + MadLib installation

PostgreSQL is a very very powerful database engine by itself, there are many advanced functions which is far beyond the standard SQL query.

The following combo are great as a starting point with the following functionality.

TimescaleDB – Time Series Database, support time based partitioning of data sharding.
https://docs.timescale.com/latest/getting-started

MadLib – a DB extension to carry out common Machine learning program within the DB.
https://cwiki.apache.org/confluence/display/MADLIB/Architecture

Here are the installation steps.

# install PostgreSQL and PGXN client
deb http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get -y install postgresql-10 libpq-dev postgresql-server-dev-10 postgresql-plpython-10 pgxnclient cmake g++ m4

# Include the TimescaleDB PPA
sudo add-apt-repository ppa:timescale/timescaledb-ppa

sudo apt-get update

# Install the TimescaleDB
sudo apt install timescaledb-postgresql-10
sudo timescaledb-tune --quiet --yes

# Install MADLib
sudo pgxnclient install madlib
sudo pgxnclient load madlib 

# Install CStore
sudo apt-get install bison flex git libreadline-dev libz-dev git libpq-dev libprotobuf-c0-dev make protobuf-c-compiler
sudo pgxn install cstore_fdw
sudo pgxn load cstore_fdw

After you installed all of them, you may play around with the example.
https://docs.timescale.com/latest/tutorials/tutorial-hello-nyc

For each newly created db, we need to enable the extensions, for example.

CStore Reference
https://www.citusdata.com/blog/2014/04/03/columnar-store-for-analytics/
https://info.citusdata.com/rs/235-CNE-301/images/Columnar_Store_for_PostgreSQL_Using_cstore_fdw_Webinar_Slides_0915.pdf

CREATE USER jimmy WITH PASSWORD 'xxxxxx';
CREATE DATABASE testdb OWNER=jimmy LC_COLLATE='C' LC_CTYPE='C' template=template0;
GRANT ALL ON DATABASE testdb to jimmy;
\c testdb
CREATE EXTENSION plpythonu;
CREATE EXTENSION madlib;
CREATE EXTENSION timescaledb;
CREATE EXTENSION cstore_fdw;
CREATE SERVER cstore_server FOREIGN DATA WRAPPER cstore_fdw;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public to jimmy;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public to jimmy;
GRANT ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA public to jimmy;

WebDAV Server on Apache

MS Office has native support of WebDAV, you can quickly setup a WebDAV server in Apache for testing, instead of spinning up the fully functioning SharePoint Server.

Here is the quick settings for setting up a Apache WebDAV server.


a2enmod dav
a2enmod dav_fs

And then add a new site in /etc/apache2/sites-available/ as followed.

<VirtualHost *:80>
        ServerName webdav.jimmysyss.com
        ServerAdmin webmaster@localhost
        ServerAlias webdav.jimmysyss.com
        DocumentRoot /var/www/webdav

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        <Directory />
                Options FollowSymLinks
                AllowOverride None
        </Directory>
        <Directory /var/www/webdav/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>

        Alias /svn /var/www/webdav/svn
        <Location /svn>
                DAV On
        </Location>
</VirtualHost>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

Meltdown and Spectre affects you?

These few vulnerabilities claims to be the most widely spread potential security issue in the last few years. M$ and Linux has provided corresponding patch in fixing it. However, the fix is not free, it usually counts for ~10% of performance reduction, the effect may be magnified in IO intensive usage.

I found my machine compiling our source slower than before by slim margin (~30 sec difference) and try to dig out the reason of the slowness.

There are ways to stop the patch in either Linux and Windows.

Linux
http://wayoflinux.com/blog/meltdown-spectre-performance

Windows
https://www.grc.com/inspectre.htm

Honestly, I am not a terrorist nor government officers, given my notebook is sitting behind the company firewall, I am safe to reclaim my PC performance. You may check that out too, but do at your own risk.

Commonly used Java libraries

Java is a classical general purposes language that can do most of the things. Java programming constructs are too primitive that people are complaining about writing redundant code, like sorting, searching, set union or intersection and etc on their own.

In fact, there are ready to use utility libraries that could save us hours in writing common logics, the only matter is the programmer attitude towards re-using others’ code or re-inventing the wheels.

Most of the utilities come from the following sources. I always search before I write code.
1. JDK
2. Apache Common
3. Spring Utilities
4. Google Guava

Here are my favourite lists of libraries.

JDK Utilities
JDK has comes with comprehensive functions for Collections(List, Set, Map) and Arrays. Most collection related functions like Sorting, Searching, Union, Swapping, Reverse orders are already supported, which means that writing 2 nested for-loop for sorting and one for loop for searching are too outdated.
java.util.Collections
https://docs.oracle.com/javase/7/docs/api/java/util/Collections.html
java.util.Objects
https://docs.oracle.com/javase/7/docs/api/java/util/Objects.html
java.util.Arrays
https://docs.oracle.com/javase/7/docs/api/java/util/Arrays.html

IO Related Utilities
When we deal with Java Streams, Reader, Writer and Files, writing buffer reading logics are too comsy and hard to get it right, like the try-catch-finally structure for Streams and Readers. IOUtils provides those static method for it.
org.apache.commons.io.FileUtils
https://commons.apache.org/proper/commons-io/javadocs/api-2.6/org/apache/commons/io/FileUtils.html
org.apache.commons.io.IOUtils
https://commons.apache.org/proper/commons-io/javadocs/api-2.6/org/apache/commons/io/IOUtils.html

String Utilities
There are String utilities comes from different libraries, they usually provide functions like substring, joining, regex matching, splitting, extraction and search & replace . Some of the functionality are overlapping, so, you need to look at the API before you start.
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/util/StringUtils.html
https://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/StringUtils.html
https://google.github.io/guava/releases/19.0/api/docs/com/google/common/base/Strings.html

Bean Utilities
The Java Reflection API is inherently hard to use and error prone, we need to handle properties accessor, access level, value getter and setters. Spring provides BeanUtils and BeanWrappers that make this kind of access much easier.
org.springframework.beans.BeanUtils
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/beans/BeanUtils.html
BeanWrapper and BeanWrapperImpl
We can create a BeanWrapper and access the property value by property name (String).
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/beans/BeanWrapper.html
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/beans/BeanWrapperImpl.html

More advanced data structure
Google Guava provides advanced data structure, like MultiSet (Count for an object occurrence), Multimap (Group objects under the same key), BiMap (Key->Value AND Value->Key mapping) and etc. You can think twice when you deal with Map or Map> which looks very complicated.
https://github.com/google/guava/wiki/NewCollectionTypesExplained

More collections utilities from Guava
Google Guava provides more advanced Collection related utilities, which is not provided by JDK Collections class, like Cartesian Products, Subset operation and etc. It may also be first steps to Java Functional programming and bridge to Java 8 Lambda expression.
https://github.com/google/guava/wiki/CollectionUtilitiesExplained

Self Reflection on IT company structure

Interesting stories to share

https://dev.to/henrylim96/reverse-engineering-cathay-pacifics-seat-selection-page-43od

I am imagining how’s CX handle this incident internally and how to avoid. CX “supposes” to have good system and control, every thing should have check and balance. Who should be responsible for this?????

Imagining there are standard in-house software development structure, different teams would have claims as followed.
Business User: IT is shit, making rubbish, charge me so much (transfer pricing). Fire them all!!! (Yes, they did, I think they deserve)

Business Analyst: I have already documented the user requirement and expectation, modifying the ticket class is not a valid use case, it should be security team responsibility, definitely not my fault.

Security Team: My responsibility is using the million dollars app scanner, network scanner, IDS (Intrusion detection system) and XYZXYZ (lots of buzz words) to do regular checking, I just know scanning, but nothing about business.

Dev Team: Such validation is not written on the specification, it makes no sense for me to implement it.

Micro-service Dev Team: This logic suppose to be validated by XXX Team, it is not my responsibility to re-validate and I am NOT TOLD TO DO SO.

Architect: (Playing fingers) It is business use case, not on my dish.

QA: BA, pls confirm(The requirement). DB Team, pls confirm. Dev Team, pls confirm. I don’t know who should I ask to confirm. I am just a test plan executor. I can be BA if I know the business well, I could be a programmer if I can code test case. This incident is definitely not my issue.

DB Team: I only deal with DB Structure and constraint.

Support / Customer Service: The phones are all ringing, the customer has fxxked us so hard. Dev Team, pls advice. BA, pls advice. DB Team, pls advice. Architect Team, pls advice.

Internal Audit Team: I am just a Business Man, knowing how to present and tender external party for auditing. I don’t really know how the system works, how could I audit to this level?

The management may claim everyone is responsible, but eventually it means no one is responsible.

It is ironic that simple script kiddie technique can break several million dollars project, and destroy the brand. I don’t think this is the only bug on the system or any other multi billions dollars organization, from banks, to hospitals, to varies online providers.

Disclaimer: Any similarity is mere coincidence

Setup IBM MQ on Ubuntu

My recently is recently integrating with a banking solution via IBM MQ, I need to install an IBM MQ for testing. Here are my steps for the installation

PS. IBM MQ 9.1 requires IPv6 for all communication, so, I fall back to IBM MQ 8.0

1. Download IBM MQ binary from IBM Website
https://developer.ibm.com/messaging/mq-downloads/

2. Follow the following link for the installation
https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.ins.doc/q008640_.htm

3. After installation, add ubuntu user (current user) to the “mqm” group

sudo usermod -aG mqm $USER

4. Up to now, the installation should be completed.

5. Setup Env and Queue Manager

There is a default MQ Installation (Installation1) setup, you can use the installation directly.

The program directory is /opt/mqm , you can find all command in /opt/mqm/bin , all subsequent execution are in that folder.

6. Create the MQ Manager

crtmqm QM1
strmqm QM1

7. Download the sample script from the following link, it will create some default queue and settings.

https://raw.githubusercontent.com/ibm-messaging/mq-dev-samples/master/gettingStarted/mqsc/mq-dev-config.mqsc

Modify the script, change the user from “app” to “ubuntu”

DEFINE CHANNEL(‘DEV.APP.SVRCONN’) CHLTYPE(SVRCONN) MCAUSER(‘ubuntu’) REPLACE

Execute the script

./runmqsc QM1 < ~/mq-dev-config.mqsc

8. MQ Server uses Linux user and user group for authentication and authorization, so, we need to assign more MQ Server privilege to Linux user group (mqm)

./setmqaut -m QM1 -t qmgr -g mqm +connect +inq
./setmqaut -m QM1 -n DEV.** -t queue -g mqm +put +get +browse +inq

9. Finally, since IBM MQ 7.5, Channel would also require authentication, but I have no idea on how it works yet, so the simplest solution is disabled that.

./runmqsc QM1
ALTER QMGR CHLAUTH(DISABLED)

PS. We should figure out how to work properly with the channel authentication.

http://www-01.ibm.com/support/docview.wss?uid=swg21577137

10. Testing with MQ Explorer

Install IBM MQ Explorer from the following link

https://www-01.ibm.com/support/docview.wss?uid=swg24021041

You can use MQ Explorer to connect to .39 MQ Queue Manager.

Build my Fusion Drive with LVM-Cache

LVM Cache is a interesting feature that I saw some web review on the web. It can speed up traditional mechanical disk with a cache partition on SSD. Essentially the concept is the same as Windows Readyboost or Fusion Drive, which the caching is controlled by firmware.

Resize the SSD

e2fsck /dev/sda1

resize2fs /dev/sda1 100000M

Use GParted to adjust the partition size

Mark Data(sdb1), Cache(sda2) and Meta(sda3) as LVM PV

sudo pvcreate /dev/sdb1 

sudo pvcreate /dev/sda2

sudo pvcreate /dev/sda3

Create Volume Group with the PVs

# Must spare some space
sudo vgcreate VG /dev/sdb1 /dev/sda2 /dev/sda3

sudo lvcreate -l 95%FREE -n data VG /dev/sdb1

sudo lvcreate -l 95%FREE -n cache VG /dev/sda2

sudo lvcreate -l 95%FREE -n meta VG /dev/sda3

Create Cache Pool and config cache mode as Writeback (improve read / write performance)

sudo lvconvert --type cache-pool --poolmetadata VG/meta VG/cache

sudo lvconvert --type cache --cachepool VG/cache --cachemode writeback VG/data

In case you are unlucky that u encounter the cache corruption. You need to execute the following commands to rebuild the cache

vgchange -a y VG
lvchange -a y VG/data
lvconvert --repair VG/data

Reference:
http://man7.org/linux/man-pages/man7/lvmcache.7.html

ActiveMQ Artemis quick start notes

I have played around with Artemis for a few days, here are the useful command I used. The exact config is stored at my github.

https://github.com/jimmysyss/artemis-config

1. Create 2 brokers with port offset
artemis create mybroker1 –port-offset 0
artemis create mybroker2 –port-offset 0

2. Testing the producer and consumer
./artemis producer –url tcp://localhost:61616
./artemis consumer –url tcp://localhost:61616

3. Enabled clustered
See jgroup.xml and broker.xml , and

4. Load balancing
See broker.xml, Add in

5. Enable SSL for HornetQ OpenWire protocol
a. Provide keystore.jks and truststore.jks
b. Config the acceptor in broker.xml

Nginx per user directory

It is common for hosting company to share the same host and create per user directory, so that every user can browse with the following url.


http://localhost/~jimmy/index.html

It can be achieved easily by nginx configuration.


sudo vi /etc/nginx/sites-available/default

Add the following code under “server” section

location ~ ^/~(.+?)(/.*)?$ {
alias /home/$1/www$2;
autoindex on;
}

Add the user to www-data group

sudo usermod -aG www-data $USER

Change the www directory under user home to mod 755, so that others can access the folder with execute right

chown 755 -R ~/www/

MiniDLNA configuration

MiniDLNA is a uPnP server that allow you to play Video, Picture and Music over LAN via IP broadcasting protocol.

I have installed it in my Raspberry Pi as a mini-server at home, to stream video to my TV and my XiaoMi TVBox.

By default, it runs with “minidlna” user, it may not be so convenient for use because it is not a usual user, I need to manually copy files to the Pi and subsequently to the folder accessible by minidlna. It is related to the user privilege issue. Funny enough, simply change on folder to 666 doesn’t solve the problem because minidlna requires to access the folder along the path from “/”

Therefore, the only solution is to configure MiniDLNA to run on another user account rather than the default minidlna user.

The minidlna.conf is as followed


# Specify the user name or uid to run as.
user=pi
....
#media_dir=/var/lib/minidlna
media_dir=/home/pi/DLNA
....
#db_dir=/var/cache/minidlna
db_dir=/home/pi/.dlnacache
....
#log_dir=/var/log
log_dir=/home/pi/.dlnalog