Author Archives: jimmy

Example of NGINX reverse proxy config

Common Reverse Proxy

upstream foundation {
    server localhost:18080;
    keepalive 8;

server {
        listen 443 ssl http2;
        listen [::]:443 ssl http2;


        root /var/www/pwa;
        index index.html;

	location /foundation/ {
		proxy_set_header Connection "";
		proxy_set_header Host $http_host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header X-Frame-Options SAMEORIGIN;
		proxy_pass http://foundation/;

	location / {
                try_files $uri $uri/ /index.html;
#		try_files $uri $uri/ =404;
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m u$

server {
        listen 443 ssl http2;
        listen [::]:443 ssl http2;

        index index.html;

   location ~ /api/v[0-9]+/(users/)?websocket$ {
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       client_body_timeout 60;
       send_timeout 300;
       lingering_timeout 5;
       proxy_connect_timeout 90;
       proxy_send_timeout 300;
       proxy_read_timeout 90s;
       proxy_pass http://localhost:8065;

   location / {
       client_max_body_size 50M;
       proxy_set_header Connection "";
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_cache mattermost_cache;
       proxy_cache_revalidate on;
       proxy_cache_min_uses 2;
       proxy_cache_use_stale timeout;
       proxy_cache_lock on;
       proxy_http_version 1.1;
       proxy_pass http://localhost:8065;

Ansible First Impression

I have tested out Ansible recently for Server admin, the experience is nice but it may take time to understand the whole eco-system.

There are a few concepts to understand before hand

Ansible uses SSH to execute Python Code in the Client machine, therefore, the Client needs to install OpenSSH and Python by default. It looks up Python 2 by default, a custom parameter is needed to instruct Ansible to use Python 3 instead. Everything makes sense if follows this direction.

Ansible Playbook is the script that drives the client machines to work, there are a lot of build in modules in Ansible. Worst case scenario is using the shell script module to write all commands. In this case, the script may only execute once, and may have side effect if execute more than once.

Ansible Collections and Roles are pre-built extensions for Ansible, a role usually serve one function, for example, managing sysctl.conf service or managing Grub. A collection is a collection of Role that may have multiple functions.

Ansible Galaxy is the marketplace of Ansible Collections and Ansible Roles. However, only those frequently downloaded scripts are useful in general. Others are just for a niche use cases.

Finally, the Ansible comes with Ubuntu 18.04 APT repo are outdated(Version 2.5), you need version 2.8 or 2.9 to be useful. You can install by installing the via installing the Ansible APT library.

sudo apt update
sudo apt install software-properties-common
sudo apt-add-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

Finally get my Hackintosh (High Sierra) working!!!!!!

After serveral years of studies and trial, I finally get a Hackintosh working with spared parts. I know the cost for a proper Mac Book Pro is far below my time cost, but it is really a great learning experience for me.

Before I start, here is my hardware list, I just purchase 2nd parts, with my old parts and my brother-in-law decomissioned parts.

Intel E3-1275 (Sandy Bridge)
Biostar B75S3E (B75 mATX board)
Asus GT1030 2GB
Crucial BX500 240G SSD

Basically it covers the following major steps. I use Hackintosh Zone High Sierra for installation.
1. Setup the BIOS according to the Hackintosh guide
2. Setup the Hackintosh with relevant settings
3. Upgarde to 10.13.6 (3 Updates)
4. Use Clover Configurator to setup the SMIBIOS, prepare for Web Drivers and enable SSDT flags for power management
5. Use Multibeast to install a bunch of Drivers
6. Install Nvidia Web Driver to enable the graphics acceleration of GT1030
7. Convert APFS and configure mount point noatime to preserve SSD wearing
8. Homebrew
9. Enjoy!!

The detail procedure is as followed

2. Setup the Hackintosh with relevant settings
I have done the following settings in the configuration for Installing the base MacOS
a. Enable Network
b. Enable USB Support for Intel 7/8/9 family USB
c. Disable NullCPUPowerManagement (enable by Default)

4. Use Clover Configurator to setup the SMIBIOS, prepare for Web Drivers and enable SSDT flags for power management
Primarily, this settings is for SSDT to enable power management in Clover Configurator. My Settings are as followed.

5. Use Multibeast to install a bunch of Drivers.
Multibeast is used to install a bunch of drivers to fit my hardware. The list is as followed.

6. Nvidia Web Drivers
Nvidia Web Driver is provided by nVidia (not Apple) to drive the latest GeForce series graphics card. Unfortunate, it supports only up to High Sierra, no Majove or Catalina. We could download here. It has to match with your MacOS version, including sub-version and patch level

7. APFS and noatime mount options
APFS is the latest file system of MacOS which support SSD Trim command. In the Niresh Hackintosh Disc, it doesn’t come with the drivers, which I cannot install MacOS on my SSD. Therefore, I need to make conversion after installation. Luckily, the installation is very simple.
1. Boot into recovery mode
2. Unmount the Disk
3. Edit => Convert to APFS
In unix file system, it has a function named atime, which will log the time for every file access. It is no big deal in Magnetic hard disk, however, it is a matter for SSD as it hugely increase the wearing of SSD. Furthermore, because it saves a write operation, the disk will also perform slightly faster, especially for compiling program
2. Reboot, check with “mount”

8. Homebrew,
/usr/bin/ruby -e “$(curl -fsSL”

There are some pitfall which I have encountered, the best recommendation is still strictly follow the tonymacx86 purchase list, and don’t even derivate a single model from it.

The pitfall are as followed.

1. Sandy Bridge CPU E3 1275 iGPU is HD P3000 which has a different hardware device ID with a standard Sandy Bridge HD3000. Inject Intel and FakeIntelGPU may work, but fail due to point 2
2. Mix of Sandy Bridge CPU + Ivy Bridge Mainboard will cause issue. Apple treats mainboard as the platform, which you need to match up with CPU and iGPU by configuration SMIBIOS. I cannot get the iGPU working and need to use a GT1030 card
3. MacOS is not happy with nVidia Maxwell and Pascal GPU out of the box, that’s the 9xx and 10xx family. They cannot run on Mojave OR Catalina right now. For High Sierra, nVidia has provide official Drivers named Nvidia Web Driver, You need to pick the right version against your current Mac Version, 10.13.0 has different driver from 10.13.6.
4. Picking a Kelper Nvidia GPU is a MUST for Mojave or Catalina. Comments from forum suggest RX570. I am wondering if a RX460 from China, which is the salvage of Crypto Mining Machine is a good deal
4. You MUST go to Recovery Mode and UNMOUNT the system volume to convert the primary partition to APFS
5. Don’t need to install the NullCPUManagement.kext. The AppleIntelCPUPowerManagement.kext should be fine and working with the Clover C-State parameters


Installing OracleXE 18 on CentOS on Proxmox VE LXC

Installing Oracle used to be a tedious and error prone operation, it takes my inexperienced colleague a week, but still cannot fix it. I have fixed the server and would make note on the procedures to install Oracle so that I won’t spend too much time on it next time.

Basically I follow the RHEL steps in the following links. CentOS on LXC is exactly the same in the following link.

I need to prepare the environment first as followed. Everything is done by “root”

  • Host Table – Hard Code the hostname in /etc/hosts, with the following line    OracleCentOS
  • Disable IPv6 – seems IPv6 doesn’t work very well in Oracle XE 18, I can only connect locally (with port forwarding) but not from another machine. 
    edit /etc/sysctl.conf , amend the following two lines at the end of filenet.ipv6.conf.all.disable_ipv6 = 1
    net.ipv6.conf.default.disable_ipv6 = 1
  • Install curl for subsequent installation steps
    yum install curl
  • Restart server to make every thing effective. 
    shutdown -r now

And then we could follow the above link “Red Hat compatible Linux distribution” section to install the Oracle

We can get an up-and-running Oracle up to now, however, it can only be connect by sqlplus, you cannot connect to it from external. It is due to the listener settings is not correct yet. So, we need to go through a few post installation configuration steps.

  • Setup the Oracle default DB
    /etc/init.d/oracle-xe-18c configure
  • Modify tnsname.ora and listener.ora, use hostname OracleCentOS and port 1521
    vi /opt/oracle/product/18c/dbhomeXE/network/admin/tnsnames.ora

tnsnames.ora Network Configuration File: /opt/oracle/product/18c/dbhomeXE/network/admin/tnsnames.ora
Generated by Oracle configuration tools.
XE =
(ADDRESS = (PROTOCOL = TCP)(HOST = OracleCentOS)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = OracleCentOS)(PORT = 1521))

  • vi /opt/oracle/product/18c/dbhomeXE/network/admin/listener.ora

      (ADDRESS = (PROTOCOL = TCP)(HOST = OracleCentOS)(PORT = 1521))

  • Restart the oracle service by the following command 
    /etc/init.d/oracle-xe-18c restart
  • You can verify port 1521 opens with the following command
    netstat -anp | grep 1521

You should be able to connect to the Oracle via IP as sysdba in SQLDeveloper

it is strange that I cannot simply use SQLDeveloper to create user, we need to modify the final SQL to create user a bit.

Therefore, what we do is to add this line in the SQL tab during create user.
alter session set "_ORACLE_SCRIPT"=true;  

PostgreSQL + TimescaleDB + MadLib installation

PostgreSQL is a very very powerful database engine by itself, there are many advanced functions which is far beyond the standard SQL query.

The following combo are great as a starting point with the following functionality.

TimescaleDB – Time Series Database, support time based partitioning of data sharding.

MadLib – a DB extension to carry out common Machine learning program within the DB.

Here are the installation steps.

# install PostgreSQL and PGXN client
deb bionic-pgdg main
wget --quiet -O - | sudo apt-key add -
sudo apt-get update
sudo apt-get -y install postgresql-10 libpq-dev postgresql-server-dev-10 postgresql-plpython-10 pgxnclient cmake g++ m4

# Include the TimescaleDB PPA
sudo add-apt-repository ppa:timescale/timescaledb-ppa

sudo apt-get update

# Install the TimescaleDB
sudo apt install timescaledb-postgresql-10
sudo timescaledb-tune --quiet --yes

# Install MADLib
sudo pgxnclient install madlib
sudo pgxnclient load madlib 

# Install CStore
sudo apt-get install bison flex git libreadline-dev libz-dev git libpq-dev libprotobuf-c0-dev make protobuf-c-compiler
sudo pgxn install cstore_fdw
sudo pgxn load cstore_fdw

After you installed all of them, you may play around with the example.

For each newly created db, we need to enable the extensions, for example.

CStore Reference

CREATE DATABASE testdb OWNER=jimmy LC_COLLATE='C' LC_CTYPE='C' template=template0;
GRANT ALL ON DATABASE testdb to jimmy;
\c testdb
CREATE SERVER cstore_server FOREIGN DATA WRAPPER cstore_fdw;

WebDAV Server on Apache

MS Office has native support of WebDAV, you can quickly setup a WebDAV server in Apache for testing, instead of spinning up the fully functioning SharePoint Server.

Here is the quick settings for setting up a Apache WebDAV server.

a2enmod dav
a2enmod dav_fs

And then add a new site in /etc/apache2/sites-available/ as followed.

<VirtualHost *:80>
        ServerAdmin [email protected]
        DocumentRoot /var/www/webdav

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        <Directory />
                Options FollowSymLinks
                AllowOverride None
        <Directory /var/www/webdav/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all

        Alias /svn /var/www/webdav/svn
        <Location /svn>
                DAV On

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

Meltdown and Spectre affects you?

These few vulnerabilities claims to be the most widely spread potential security issue in the last few years. M$ and Linux has provided corresponding patch in fixing it. However, the fix is not free, it usually counts for ~10% of performance reduction, the effect may be magnified in IO intensive usage.

I found my machine compiling our source slower than before by slim margin (~30 sec difference) and try to dig out the reason of the slowness.

There are ways to stop the patch in either Linux and Windows.



Honestly, I am not a terrorist nor government officers, given my notebook is sitting behind the company firewall, I am safe to reclaim my PC performance. You may check that out too, but do at your own risk.

Commonly used Java libraries

Java is a classical general purposes language that can do most of the things. Java programming constructs are too primitive that people are complaining about writing redundant code, like sorting, searching, set union or intersection and etc on their own.

In fact, there are ready to use utility libraries that could save us hours in writing common logics, the only matter is the programmer attitude towards re-using others’ code or re-inventing the wheels.

Most of the utilities come from the following sources. I always search before I write code.
1. JDK
2. Apache Common
3. Spring Utilities
4. Google Guava

Here are my favourite lists of libraries.

JDK Utilities
JDK has comes with comprehensive functions for Collections(List, Set, Map) and Arrays. Most collection related functions like Sorting, Searching, Union, Swapping, Reverse orders are already supported, which means that writing 2 nested for-loop for sorting and one for loop for searching are too outdated.

IO Related Utilities
When we deal with Java Streams, Reader, Writer and Files, writing buffer reading logics are too comsy and hard to get it right, like the try-catch-finally structure for Streams and Readers. IOUtils provides those static method for it.

String Utilities
There are String utilities comes from different libraries, they usually provide functions like substring, joining, regex matching, splitting, extraction and search & replace . Some of the functionality are overlapping, so, you need to look at the API before you start.

Bean Utilities
The Java Reflection API is inherently hard to use and error prone, we need to handle properties accessor, access level, value getter and setters. Spring provides BeanUtils and BeanWrappers that make this kind of access much easier.
BeanWrapper and BeanWrapperImpl
We can create a BeanWrapper and access the property value by property name (String).

More advanced data structure
Google Guava provides advanced data structure, like MultiSet (Count for an object occurrence), Multimap (Group objects under the same key), BiMap (Key->Value AND Value->Key mapping) and etc. You can think twice when you deal with Map or Map> which looks very complicated.

More collections utilities from Guava
Google Guava provides more advanced Collection related utilities, which is not provided by JDK Collections class, like Cartesian Products, Subset operation and etc. It may also be first steps to Java Functional programming and bridge to Java 8 Lambda expression.

Self Reflection on IT company structure

Interesting stories to share

I am imagining how’s CX handle this incident internally and how to avoid. CX “supposes” to have good system and control, every thing should have check and balance. Who should be responsible for this?????

Imagining there are standard in-house software development structure, different teams would have claims as followed.
Business User: IT is shit, making rubbish, charge me so much (transfer pricing). Fire them all!!! (Yes, they did, I think they deserve)

Business Analyst: I have already documented the user requirement and expectation, modifying the ticket class is not a valid use case, it should be security team responsibility, definitely not my fault.

Security Team: My responsibility is using the million dollars app scanner, network scanner, IDS (Intrusion detection system) and XYZXYZ (lots of buzz words) to do regular checking, I just know scanning, but nothing about business.

Dev Team: Such validation is not written on the specification, it makes no sense for me to implement it.

Micro-service Dev Team: This logic suppose to be validated by XXX Team, it is not my responsibility to re-validate and I am NOT TOLD TO DO SO.

Architect: (Playing fingers) It is business use case, not on my dish.

QA: BA, pls confirm(The requirement). DB Team, pls confirm. Dev Team, pls confirm. I don’t know who should I ask to confirm. I am just a test plan executor. I can be BA if I know the business well, I could be a programmer if I can code test case. This incident is definitely not my issue.

DB Team: I only deal with DB Structure and constraint.

Support / Customer Service: The phones are all ringing, the customer has fxxked us so hard. Dev Team, pls advice. BA, pls advice. DB Team, pls advice. Architect Team, pls advice.

Internal Audit Team: I am just a Business Man, knowing how to present and tender external party for auditing. I don’t really know how the system works, how could I audit to this level?

The management may claim everyone is responsible, but eventually it means no one is responsible.

It is ironic that simple script kiddie technique can break several million dollars project, and destroy the brand. I don’t think this is the only bug on the system or any other multi billions dollars organization, from banks, to hospitals, to varies online providers.

Disclaimer: Any similarity is mere coincidence

Setup IBM MQ on Ubuntu

My recently is recently integrating with a banking solution via IBM MQ, I need to install an IBM MQ for testing. Here are my steps for the installation

PS. IBM MQ 9.1 requires IPv6 for all communication, so, I fall back to IBM MQ 8.0

1. Download IBM MQ binary from IBM Website

2. Follow the following link for the installation

3. After installation, add ubuntu user (current user) to the “mqm” group

sudo usermod -aG mqm $USER

4. Up to now, the installation should be completed.

5. Setup Env and Queue Manager

There is a default MQ Installation (Installation1) setup, you can use the installation directly.

The program directory is /opt/mqm , you can find all command in /opt/mqm/bin , all subsequent execution are in that folder.

6. Create the MQ Manager

crtmqm QM1
strmqm QM1

7. Download the sample script from the following link, it will create some default queue and settings.

Modify the script, change the user from “app” to “ubuntu”


Execute the script

./runmqsc QM1 < ~/mq-dev-config.mqsc

8. MQ Server uses Linux user and user group for authentication and authorization, so, we need to assign more MQ Server privilege to Linux user group (mqm)

./setmqaut -m QM1 -t qmgr -g mqm +connect +inq
./setmqaut -m QM1 -n DEV.** -t queue -g mqm +put +get +browse +inq

9. Finally, since IBM MQ 7.5, Channel would also require authentication, but I have no idea on how it works yet, so the simplest solution is disabled that.

./runmqsc QM1

PS. We should figure out how to work properly with the channel authentication.

10. Testing with MQ Explorer

Install IBM MQ Explorer from the following link

You can use MQ Explorer to connect to .39 MQ Queue Manager.