Category Archives: Uncategorized

RDBMS vs NoSQL vs NewSQL

SQL and traditional RDBMS is inevitably a key surviving skill for every programmer. SQL is good given that it is more or less an universal standard in database, or when we want to interact with any database. RDBMS is good because they provide ACID guarantee for programmer, which drastic simplify programmer life especially in web environment.

However, RDBMS suffers from scalability and concurrency problem when it comes to web scale. The common 1 master + N slaves or data sharding technique only postpone the problems by several times. It also posts a few limitations like Read-Write ratio and the partition key has to be carefully selected, which means the designers have to aware of the limitation.

Some people start moving on to NoSQL, whether it is NO SQL or NOT ONLY SQL is debatable. However, from an engineering point of view, NoSQL is a solution to a specific problem but not a silver bullet.

In general, NoSQL can be further divided into the following categories.
– Graph Database – Neo4j
– Document Base Database – MongoDB
– Key Value Database – Redis / Memcache
– Column Oriented Database – Cassandra
– Time Series Database (Extension) – Riak DB / OpenTSDB

Each of them are solving a particular business model or use cases. For example, Graph DB are used to handle parties and relationships very efficiently. Document Base DB can handle hierarchy data better than RDBMS. We usually trade scalability for giving up Transaction capability.

The famous CAP theorem (Brewer’s theorem) describes the situation that Consistency, Availability and Partition Tolerance are mutually exclusive, we can only pick two out of three. Each NoSQL DB and RDBMS are following this rules with NO EXCEPTION.

SQL is still widely used as of today, since many software are not really web scale or the management doesn’t know or want to pay the cost for web scale. So, people are dreaming to have SQL & RDBMS capabilities while having the NoSQL scalability? It becomes the goal for NewSQL. Google (The God, again) has published a paper for the concept of Spanner, which is now in production in Google Cloud Platform.

https://research.google.com/archive/spanner.html

In short, it tries to detach the TX manager and the underlying Storage Manager, so that for each query or update, the TX Manager can acquire the relevant Storage managers. Since the Storage is handle locally, the storage itself is usually bigger in size (64MB) and can be distributed. It fits perfectly in AWS S3 or Google Cloud Storage. However, as it involves Network operations, the overall performance should be slower than a local DB with smaller dataset.

There are other local implementations, like VoltDB or CockroachDB.

SailsJS Hello World

Sails JS is a full MVC framework that build on Express and io.js. It looks easy to use and I have tried to incorporated different functionalities in it.

You may find my hello world in the following links

https://github.com/jimmysyss/sails-hw

I have included the following in the projects.

1. Passport.js
2. Swaggers.js
3. React + Webpack
4. Sequelize

I have also configured a few settings in the Sails JS Project.

1. CSRF
2. Winston

Notes for Websphere and DB2 on Linux

WebSphere

WebSphere has to been installed with IBM Installation Manager.

http://www-01.ibm.com/support/docview.wss?uid=swg24043138

After starting it, it will ask for the repository, you may need the following repository

Version 9.0: https://www.ibm.com/software/repositorymanager/V9WASILAN
Version 8.5: https://www.ibm.com/software/repositorymanager/V85WASDeveloperILAN
Version 8.0: https://www.ibm.com/software/repositorymanager/V8WASDeveloperILAN

You need to make sure you install the optional JDK_1.7.0 or JDK_1.7.1. In WebSphere, it uses a patched JDK.

DB2 LUW

Creating Database

db2 create database oms;
db2 activate database oms;
db2 connect to oms;
db2 connect to database oms user db2inst1 using password

Drop Database

In some case, there may be connection to the existing schema, so, it is easier to restart the DB before droping the tables

db2stop force
db2start
db2 drop database oms

Backup Database

db2look -d oms -z db2inst1 -e -o oms.ddl

Restore Database

db2 -vmf oms.ddl

Extract single table to ixl (IBM Information Exchange Format)

db2 “export to aa_user.del of del SELECT * FROM AA_USER”

Import Data from IXL

db2 load from /tmp/department.ixf of ixf replace into department2

Create Tablespace with larger page size

db2 create bufferpool bp32k pagesize 32K
db2 create tablespace data32k pagesize 32K bufferpool bp32K

Set rev_info for Hibernate envus

db2 alter table rev_info alter column rev set generated always as identity

Table space

db2 create bufferpool bp32k pagesize 32K
db2 create tablespace data32k pagesize 32K bufferpool bp32K
db2 CREATE SYSTEM TEMPORARY TABLESPACE STB_32 PAGESIZE 32K BUFFERPOOL BP32K

Export CLOB or BLOB
db2 “export to TBL_XXX.ixf of ixf lobs to ./ SELECT * FROM TBL_XXX”

Download Stocklist for SGX and HKEX

Complete Stock List are one of the market data product for both exchange, we may not be able to get it for free.

However, if you make use of some hacking technique, you may be able to retrieve the list

HKEX

wget https://www.hkex.com.hk/eng/market/sec_tradinfo/isincode/documents/isino.xls

SGX

curl 'https://sgx-premium.wealthmsi.com/sgx/search' -H 'Cookie: _gat=1; _ga=GA1.1.275393693.1483093908' -H 'currency: sgd' -H 'Origin: https://sgx-premium.wealthmsi.com' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.8,fr;q=0.6,zh-TW;q=0.4' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/55.0.2883.87 Chrome/55.0.2883.87 Safari/537.36' -H 'Content-Type: application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'cache-control: no-cache' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'Referer: https://sgx-premium.wealthmsi.com/index.html' --data-binary '{"criteria":[]}' --compressed > sgx_stock.json

ActiveMQ Network of Brokers with HA

ActiveMQ has two mode of configuration, a Network of Broker mode and HA mode. Network of Brokers help in distribute the loading among the network members. It is done by a client randomly connect to one of the network member, the member is responsible for resolving queue content from the producer queue.

In HA mode, they are using either a locking file (KahnDB) or locking DB (DB exclusive lock) approach. It is using a master-slave approach. When the master obtains the lock, others turns to slave mode and regularly poll for the availability of the lock. If the master goes down, the lock will be released and others will race for it.

I have tested a setup with a network of two members, and each member has the corresponding HA master-slave pair.

The configuration can be found in the following GitHub

https://github.com/jimmysyss/activemq-master-slave-settings

To test the ActiveMQ, we can use ActiveMQ producer and consumer command to test with it.

xxxxxxxxxxxx

xxxxxxxxxxxx

 

Some tricks in SSH

In Ubuntu, it is lack of unified Remote Desktop Solution like Windows. There are many other solutions VNC and FreeNX.

X11 may come to rescue in case that you need to urgently connect to a Linux machine

ssh -X -C [email protected]

We can use SSH as a Sock Proxy server, which is so called VPN for the poor.

ssh -D 5051 -C [email protected]

Getting Logitech Optical Marble Mouse works in Ubuntu

Marble mouse doesn’t comes with a wheel, which makes it not quite usable in modern OS. By following the steps, you will add a button mappings for the little left button(8) so that it can emulate the middle button and emulate Wheel Button

Add a new file /usr/share/X11/xorg.conf.d/50-marblemouse.conf

The content is as followed.

Section "InputClass"
    Identifier      "Marble Mouse"
    MatchProduct    "Logitech USB Trackball"
    MatchIsPointer  "on"
    MatchDevicePath "/dev/input/event*"
    Driver          "evdev"
    Option          "SendCoreEvents" "true"

    #  Physical buttons come from the mouse as:
    #     Big:   1 3
    #     Small: 8 9
    #
    # This makes left small button (8) into the middle, and puts
    #  scrolling on the right small button (9).
    #
    Option "Buttons"            "9"
    #Option "ButtonMapping"      "1 8 3 4 5 6 7 2 9"
    Option "ButtonMapping"      "1 9 3 4 5 6 7 2 8"
    Option "EmulateWheel"       "true"
    #Option "EmulateWheelButton" "9"
    Option "EmulateWheelButton" "8"

EndSection

Install Docker on Ubuntu

The installation procedure is extracted from Docker official homepage, just for my own reference.

1. Download and install the GPG Key

$ apt-get update
$ apt-get install apt-transport-https ca-certificates
$ sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D

2. Add the following line to /etc/apt/sources.list.d/docker.list

deb https://apt.dockerproject.org/repo ubuntu-trusty main

3. Install from Apt

$ sudo apt-get update
$ sudo apt-get install linux-image-extra-$(uname -r)
$ sudo apt-get install linux-image-generic-lts-trusty
$ sudo apt-get install docker-engine
$ sudo service docker start
$ sudo docker run hello-world

4. A docker user group is created, we add a general user to this group, the user can control docker.

$ sudo usermod -aG docker ubuntu
$ shutdown -r now

Aircrack-ng for WEP

There are more than enough Aircrack-ng tutorial on the web. I just put down the command to use for later reference here

Start wlan0 monitor mode

[email protected]:~# airmon-ng start wlan0
Found 4 processes that could cause trouble.
If airodump-ng, aireplay-ng or airtun-ng stops working after
a short period of time, you may want to kill (some of) them!

  PID Name
 2188 dhclient
 2287 NetworkManager
 3152 wpa_supplicant
 3177 dhclient

PHY	Interface	Driver		Chipset

phy0	wlan0		ath9k_htc	Atheros Communications, Inc. AR9271 802.11n
		(mac80211 monitor mode vif enabled for [phy0]wlan0 on [phy0]wlan0mon)
		(mac80211 station mode vif disabled for [phy0]wlan0)


[email protected]:~# 

Looking at which WIFI in your area

[email protected]:~# airodump-ng wlan0mon

RESULT NOT SHOWN HERE, IT EXPOSES MY LOCATION INFORMATION

Mark down the BSSID, Channel and ESSID, in particular, only WEP can be cracked.

airodump-ng --channel <channel> --bssid 00:09:5B:D7:43:A8 --write <file_name> <interface>

You will get a bunch of files, we are only interested in the “cap” file

sudo aircrack-ng <file_name>.cap

DONE….

Running PostgreSQL on Windows with Non-Privileged Account

Developers always need to test on different Database like MySQL, Oracle and PostgreSQL. It is too heavy to run them as a deamon. I prefer to run them in a no-install and portable way. I recently found and tested to run on non-privileged account, meaning that it can be portable

First, Download No-Install version of PostgreSQL

http://www.enterprisedb.com/products-services-training/pgbindownload

Extract it to a Folder, mine is C:\JimmyWork\Development\pgsql

PgSQLonWin

Create a folder “data” under pgsql

Open Command Prompt, Init the DB with this command and type the password

C:\JimmyWork\Development\pgsql>bin\initdb.exe -D data -A password -W -U postgres

Start the DB Server with this command

C:\JimmyWork\Development\pgsql>bin\postgres.exe -D data

There is a PGAdmin III in bin folder, you can use that to access the DB Started. Configure the DB as followed.

PgSQLonWin2

Usually we will create user dedicated for a DB as followed.

CREATE USER mydatabaseuser WITH PASSWORD ‘[email protected]’;
CREATE DATABASE mydatabase OWNER=mydatabaseuser ;
GRANT ALL ON DATABASE mydatabase to mydatabaseuser ;

Enjoy!!