收工交通消息Chrome Extension

icon

相關連結

Chrome Store page: https://chrome.google.com/webstore/detail/收工交通消息/amnocbjeifmlcogkicidpemmfjeimhdf

Github: https://github.com/airicyu/traffic-news-bot

介紹

這是一個關於收工時間交通消息notification的chrome extension。(只限於香港)

當你設定了收工時間之後,在星期一至五每日收工前的一小時內,這個工具就會把最新的交通消息更新推送給你。 你可以在設定頁面選擇是否推送,或只選擇推送有特定tag的交通消息。

另外,我們在顯示頁面也加入了Google map live traffic及運輸署全港主要道路交通情況的link讓你可以快速地看到現在的香港最新的路面交通情況。

 

使用方法

睇消息

裝左之後,右上plugin果度有依個extension既icon。 你可以click個icon打開popup page,就會睇到即時交通消息。

設定放工時間,等佢識早一粒鍾幫你去Check特別交通消息

popup page右上有”選項”,你click入去可以set你個放工時間(預設18:00)。 之後佢就會識計岩你放工前一小時內幫你check交通消息。
(預設下午6點收工)

Enable/Disable消息推送

如果你不喜歡消息推送,也可以在選項頁面disble這功能。
(預設enable消息推送)

關注特定交通消息的tag

如果你只想關注特定交通消息既tag (e.g. 吐露港公路 及 荃灣區),你可以去選項頁面的推送消息過濾設定。 首先取消”包括所有消息”,然後再click “吐露港公路” 及 “荃灣區”。
(預設是”包括所有消息”,會把所有消息都包括在內)

 

Screenshot

Popup頁面:

Popup頁面

選項頁面:

Push Notification頁面

Push Notification頁面:

Push Notification頁面

 

My Sharing about a Workshop for Building Serverless Application

Recently I participated a workshop about building serverless application. And this is my feedback sharing.

What is built in the workshop?

More detail to say, the workshop is actually let developers to have a taste on using a serverless cloud solution to implement a simple web application.

The web application is a kind of simple web page for polling lunch food.

 

Usual way: Web server with Backend

Usually to build such such web page, we may need a backend web server which serve the web page. And the backend may also need handle user login service, and connect with database for access the polling record data as well.

 

The “Serverless” way: Delegate backend to “backend-as-a-service” cloud

What means by the “serverless” in the workshop context is “no backend”. It means that developers may no need to handle/worry about the backend part but only focus on building the frontend web page. Hence, in the workshop, I just wrote some static HTML page and static frontend JS script only. (i.e. C Drive上網大法)

But…How can it be “no backend”????

Actually there is still kind of “backend” but it is delegated to the serverless cloud solution (“backend-as-a-service”). Hence, The backend services are hosted on cloud and services exposed as APIs.

And developers(e.g. me) may just writing static frontend page which invoking the cloud APIs in order to use login service and access the polling record data directly.

 

But My Concerns …

However, after the workshop I have some concerns about such serverless/backend-as-a-service technology.

 

1) Transparent API key in frontend

In the above simple sample which I played in the workshop, the backend APIs are requiring an static API key to call. But the API key is put in the static frontend page which is transparent to end user. (I can easily see it with firebug)

If the API key is required to call backend service or even accessing cloud database records directly, exposing the API key may introduce security problem if not carefully handled the doors.

My conversation with their engineer

I asked their engineer about my security aspect concerns. And I further ask them if they have aware of it, would it be a factor which pushing such technology to be limited on some private/internal website solutions instead.

The engineer answer me that the webpage in workshop is just for tasting. For other platform like mobile app, the source code is not visible to end user and developers may even use ProGuard to obfuscate the byte code.

Then I didn’t further talk on the questions because I feel they have no way to resolve the security issues.

Andoird app can be cracked to access APK file. And hackers can decompile APK to get source code. While ProGuard may only slow down how hackers do reverse engineering but it cannot prevent them to do so. So the problem is still there.

 

2) Potential XSS issue

During the workshop I wrote HTML page in local as C-drive-file (i.e. C Drive上網大法), but their “backend-as-a-service” APIs are on their cloud website domain.

Normally such cross domain Ajax call is blocked by browser to prevent webpage running cross site script (i.e. XSS).

Such browser security restriction can be resolve by defining Cross-Origin Resource Sharing (CORS) HTTP headers. And that is what they did.

It is OK if you don’t understand what I said in above paragraph. The fact is that, their cloud service APIs are allowing any website to access them.

What is the potential problem? Of coz … XSS attack.

If an end-user didn’t logout the “serverless” webpage and then continue browsing other websites, there is a chance which other dirty website may try invoking the cloud service APIs to hack something.

 

What did I learn?

Through the workshop I tasted the serverless/backend-as-a-service technology which is really fancy for developers to build webs without concerning much about the backend part.

However, as I stated, security is a very important concerns. And developers playing such technology must have awareness on the security aspects. Developers should ask themselves about:

Where are the back doors? Is the back doors acceptable? Or shall we use workarounds to cover the back doors? Or is it worth to take the tradeoff to use such technology after our evaluation?

 

What I have learnt from other BAAS solutions?

From time to time, actually I heard about some other technology concepts which may share some similarity of technology I tried in the workshop.

For example, I have ever heard of some other backend-as-a-service solutions provided by vendors. And they may share some security concerns which I mentioned above.

I have also ever heard of some vendor solution which directly turning the “Database with tables, schema, records” into REST API. And the solution may directly exposing the REST API to external systems. (Database-as-REST-API?)

It may be not that promising as expected although it firstly sounds good because:

1) Simply transforming Database schema into REST API definition may producing some un-developer-friendly naming problems which may not be developer friendly. And the DB schema may not be a developer-friendly representation of data structures for API consumer to understand as it is not intended for such purpose. Actually, usually a common problem is that the DB schema is too complicated and bulky that external domain consumer developers cannot understand how to directly consume it. Exposing database as REST API would not solve such problem.

2) Database has transactions. However, stateless is one of the philosophy of REST API. REST API Resources would usually not providing the same level of transaction controls. So such Database-as-REST-API approach may either broke the Database transaction functionality, or turning the REST API into something violating REST API philosophy.

(Off topic too much….so I would stop here)

想想你在年輕的時候

我有時幾鍾意接觸d freshman developer,因為岩岩出黎社會年輕有幹勁。
反而有時做多幾年野之後人好易滑晒瓦。

人在社會,either是環境把你塑造成環境的形態,or你去反影響﹑反塑造環境的形態。

當人們身處一間公司或一個環境,而人們把自己的視野與界限限定在公司裡面,其實有可能在三幾年後你就會變成了公司/環境/社會的形態。
甚至當初你有些理念而堅持反對一些東西,到最後你變成了你所反對的東西。「限制」變成了你不知不覺習慣了的一部份。

有時會聽見人們說「想去學乜乜乜」。
其實如果有心學些什麼,網上﹑世界大把resources。
我自己也不會有一種一定要在公司裡學習什麼什麼的概念,因為那是自我設限。
面向世界﹑擴闊境界,是獲益更多的。

有時我會跟一些比較fresh的同事在公事之後傾多兩句公事。
了解一下他們平時做事上有沒有遇到過些什麼問題?有沒有什麼想法可以令d事做好一點?

年輕人,會有理據地complain什麼什麼的,很好,fast feedback,能把事更好的改進。
年輕人,有時理想主義一點,是好的,值得我去反思自己是否不知不覺放棄了一些價值。
我自己希望能繼續用「年輕人」的心態做人,另一方面當面對真的後生年輕人,我也希望能幫助到他們,不要令他們滑瓦。

REST mindset

我真係見過,好多人做REST API,做到….

講REST API其實一定要知resource既概念。
我以前真係見過,有developer係用個Java interface class(util service果種)黎做REST resource。(我study過擺明唔係intended design)
點解?
因為好多developer做野唔會深究去諗點解,只要駁得通d library,call到,做到requirement,咁就OK。
佢地so called係做REST,但其實個mindset只係做RPC而已。

但其實做REST API最重要係個mindset。

點解要做REST API?
係因為對consumer來說簡單易用,低使用門檻,低學習門檻。
所以REST API最核心精神就是consumer oriented。
consumer oriented衍生出來的,就是借用HTTP protocol method & status去做Resource及CRUD。

其實真係可以問問果堆developer,佢地有冇諗過咩叫REST?點解要做REST?點樣先係consumer oriented?
如果冇,其實可以由今日開始問下自己多d依d問題。

面對API這東西,在「environment」裡面,對我來說最大的問題從來不是什麼tools,也不是business什麼,而是人們的mindset與culture。

[it學術研究] 熱血時報節目like/分享/留言數字之謎

[itdog學術研究][熱血時報節目like/分享/留言數字之謎]
 
從熱血時報網頁上看到《建國神話XXXXXX》的facebook數據,是「Like:1474 分享:1474 留言:34」
由於like及分享的數目永遠一樣,所以我自己都覺得好奇怪,所以我自己就去了驗證一下。
 
從網頁的ajax request上查找到這條GET request:
 
 
這一條request,是熱血時報call facebook的API拿取數據,這數據是來自facebook的。
而其response的結果是:
 

 

 
其response中的1474與34,正正是like/分享及留言的數目。
 
我修改了一下request:
 
 
得出的結果如下:
 

上面的數字解釋:
359: 是熱血時報原本的那個post的like count (哈哈﹑嬲嬲那些不計作like)。
19: 是熱血時報原本的那個post的comment數目。
 
至於1474與34…
 
1474那個就是平時網頁上看到Facebook like button上面的那種”1.4K people like this.”的那個數字。
我在facebook查找一下就可以知道這數字包括了什麼。
 
What makes up the number shown next to my Like button?
The number shown is the sum of:
1) The number of likes of your URL
2) The number of shares of your URL (this includes copy/pasting a link back to Facebook)
3) The number of likes and comments on stories on Facebook about your URL
 
即是原本的post的like及comment,以及這條URL在facebook上有幾多人share,以及這些post所衍生出來的like及comment,所有總數加起來的數字。
所以嚴格來說,那不是真正的「like」數目,也不是真正的「share」數目,而是facebook把所有engagement加起來的數字。
 
至於34個comment,也應該是類似的把share的數目aggregate出來的,至於實際如何計算我就未能在facebook上找到。
我也數不到如何才count到34。
 
另外,我試過share/comment,但無論是private還是public post,數字都沒有update到,所以那些value可能是cached value或Schedule update value。(未必update)
 
—————————————
 
結論:
網頁上like及share及comment的那些數字(1474, 34)是來自facebook,這數字不是假的。
不過熱血時報把那些數字labelled as 「Like:1474 分享:1474 留言:34」,其實某程度上算是有點誤導(可能是programmer之無心之誤)。
 
熱血時報原本的那個post的like及comment數目分別是359及19。(不計再share出去的post所衍生的數字)
嚴格來說,1474是engagment(參考上面)
而like及share的各自數字未能得知,但計埋衍生數字,總like數必定多於359及少於1474。(未必得知實際數目)

NodeJs 8

Node.js 8: Big Improvements for the Debugging and Native Module Ecosystem

NodeJs早兩日出左version 8了。(10月就Long Term Support)
之前Node@7.6.0+ 先有既async/await之類到時都有得用到。(very nice features)

睇nodejs@8 release notes好似冇乜其他野係可能impact到application developers。
底層engine勁左/performance improve,又或者nodejs standard native API之類果d….都未必好直接影響到application developers。
不過nodejs@8 個係跟npm@5,而npm@5開始npm install唔駛”–save”佢自動default幫你save,依個可能要留意下囉。

另外Nodejs@8會有AsyncHook API,”Continuation-Local Storage”之後會改底層implementation用AsyncHook API。
如果你係用開”Continuation-Local Storage”既,到第時轉上Nodejs@8時就可以要試下o唔ok。
“Continuation-Local Storage”類似Java既thread local,都應該common野,所以都有可能間接impact到application developers。

[Article sharing] How Reddit do View Counting?

View Counting at Reddit

This is quite a good article talking about how Reddit do view counting.

 

 

The problem has these following requirement:

  • Counts must be real time or near-real time. No daily or hourly aggregates.
  • Each user must only be counted once within a short time window.
  • The displayed count must be within a few percentage points of the actual tally.
  • The system must be able to run at production scale and process events within a few seconds of their occurrence. (Remarks: Reddit is No.8 in global visit count)

Actually such problem is categorized as “cardinality estimation problem“. (https://en.wikipedia.org/wiki/Count-distinct_problem)

 

 

A naive implementation of this solution would be to store the unique user set as a hash table in memory, with the post ID as the key.
However, it is not practical because several popular posts have over one million unique viewers and the memory & CPU usage for such solution would be too costly.

Turn out the engineers in Reddit solve it by using a combination of two algorithms for different scaling level:
1) Linear probabilistic counting
2) HyperLogLog(HLL)-based counting

Both of such algorithms used some tricky magic so that they can use extremely little memory to do the counting. (e.g: count 1M IDs using just 12 KB space)
I would not go through the details of the magic box but if you are interested you can see this article: Big Data Counting: How to count a billion distinct objects using only 1.5KB of Memory .

 

Some people had done a demo of the different counting techniques by counting the number of distinct words in all of Shakespeare’s works using three different counting techniques.

And the result is below:

Counter Bytes Used Count Error
HashSet 10447016 67801 0%
Linear probabilistic counting 3384 67080 1%
HyperLogLog 512 70002 3%

 

 

Next time if you encountered unique item counting estimation problem, you may consider using the “Linear probabilistic counting” and “HyperLogLog” techniques to help you solve the problem.

rest-in-contract – nodejs module for REST API Contract server

rest-in-contract

Project Page

Project Status

Currently, the project is in beta version (v0.x.x).

The basic Contract Server module is done to support basic usage of API Contract stubbing & testing. But some builtin feature is not done yet. (e.g: Suppoting more middleware functions in the contract script)

Since it is still beta version, we are not finalized the v1.0 in-the-box features yet.

Roadmaps

  • Add Unit tests
  • Update documents
  • Database Storage
  • Authentication
  • Support Plugins
  • Java/nodejs test integration client
  • Study on integration with Swagger

What is rest-in-contract

rest-in-contract is a product to let you embrace Consumer-driven contracts. It is REST in nature so that it fits for integrating with all kind of programming languages. For more detail about Project rest-in-contract, you may have a look in our Project rest-in-contract’s Homepage for detail introduction.

Slideshare: Basic Concepts & Flows

 


Samples

Hello world

Starting server:

Project rest-in-contract

Project rest-in-contract

Project Page

Related Projects

 

What is rest-in-contract

Consumer-driven contracts

rest-in-contract is a product to let you embrace Consumer-driven contracts. It is REST in nature so that it fits for integrating with all kind of programming languages.

Story for REST API providers/consumers in Consumer-driven contracts

For REST API providers:

REST API providers can write API contracts to describe their REST API request/response formats. They can then use the contracts to do contract testing against their API implementations.

For REST API consumers:

REST API consumers can use the API contracts to setup stubs for local testing or drafting of API contracts.

 

Slideshare: Basic Concepts & Flows

 

How does rest-in-contract different from other Consumer-driven contracts solution?

REST in nature, cross language, easy integration

There are many Consumer-driven contracts solution existing but many of them are SDK libraries or embedded solution for stubbing or testing which may fixed in a certain language. rest-in-contract is designed in a prespective that we do not want a language fix-in solution.

rest-in-contract is a node modeule to setup a lightweight agent server which API providers/consumers can kick to start in their environment easily. No matter doing contract testing(For provider) or API stubbing(For consumer), you can always do them by calling rest-constract agent server’s REST API. That’s why rest-in-contract is a cross language solution for Consumer-driven contracts.

Thanks for REST in nature, it is very easy to do integration with DevOps. No need maven or gradle build. You just need node.js(v7+) installed in your environment to kick the server. All later interactions are REST API call which you can call with curl or any other HTTP client tools.

Contract as file

Some Consumer-driven contracts solutions may let you wiring stub servers by SDK methods. Hence, the contract is writen as embedded code. Such way has difficulties for supporting different kind of programming languages.

Instead, we think that contract should be defined in a less coupling way that can be separated from your business logic codes.

The contract in rest-in-contract is described in JS script format which exporting an Contract object. We supporting some middleware function call in the contract. It also support regular expression, jsonpath etc.

An example of contract file would be like this:


 

Although the contract file is written in javascript in syntax, but you can just treat them as general files and stored in your projects. It is because that your code/application would never necessary to directly interact with the contracts. You can always pass them to the Contract Server to let it do its job.

 

What are the possible architecture configurations of rest-in-contract?

Architecture Components

Contract Server is a server instance which supporting Contract testing and stubbing by REST API. It is typically storing and reading contracts in local storage.

Contract Agent Server is actually a Contract Server. The different is that it read contracts from remote contract repository instead of local storage.

Architecture Configurations

We imagined that that there may be two architecture configurations of using rest-in-contract.

  1. Centralized Contract Server architecture

In this mode, there would be a centralized Contract Server which serving all API provider and consumers. It is the centralized storage server for persisting all contracts in database.

API providers & consumers would setup their own Contract Agent Server in their own environment which get the contracts remotely from centralized Contract Server through REST API. Then they would use their local Contract Agent Server to do contract testing or stubbing.

  1. Decentralized Contract Server architecture

In this mode, API providers would keep contracts in their own way. For examples, if their application is on Github, they may put the contracts under a folder in their source codes. API providers can kick Contract Server in local environment to do contract testing.

On the other side, API consumers can checkout the project from Github in order to get the API contracts. Then API consumers can kick Contract Server in local environment to do stubbing.

 


 

Story walkthrough

Beginning of the Story: John(API Provider) wrote an application Foo which provide REST API.

John wrote an application “Foo” which the base application URL is “http://example.com/foo“. It has a version path “/v1.0”. And the API endpoint url is “/hello”.

The API has such format:

Request:

Response:

The request body has an attribute name with a string value.

 

Next Story: Mary(API Consumer) is writing an application bar which want to consume API from John’s API.

Mary want John to enhance his API to include a new integer attribute “age” in request and output it in response like this:

Request

Response:

Hence, Mary and John have a discussion and drafted an API contract like this:


 

This single contract would be used by both John and Mary. For John, he would use this contract to do contract testing against its implementation. For Mary, she would use this contract to generate stub for local testing.

To support both use case of contract testing and stubbing, they need to define value(stub(...), test(...)) in the contract. The meaning of value(stub(...), test(...)) in contract is that, the certain values which would be used for generating stub and contract testing are different.

Why different values for stubbing and testing?

For Mary, she wants stub. The stub would accept any “name” attribute which is in [a-zA-Z ]* pattern. It means that Mary can send a request to the stub with “name” set as “Susan” or “Sam” or any other valid names…… And the response should correctly showing the same name in the request. The “name” attribute should support a flexible value so that Mary can test more dynamically instead of a always hard coded value. Hence, it is represented by a regular expression pattern regex("[a-zA-Z ]*").

For John, he wants API test. The API test just need to define a test value which is used for testing. (Actually he can use regular expression pattern as well, but here is just for demo) Hence, a hardcoded test value “John” is used. And the response value is also hardcoded test value.

 

Next Story: John use the contract to generate testing endpoint to test against the API contract

Firstly, John start a local Contract Server with port 8000.

And then he create(register) an App in Contract Server by this REST API call:

Assume the app ID is “80a69a44-3f3b-48c1-a7d1-b34b89117e75”.

For this app, John create(register) an App version “2.0” in Contract Server by this REST API call:

And then John create(register) the API contract in Contract Server by this REST API call:

(Assume the contract ID is “94923fbd-9092-4a46-ad65-0d8a2e2f551e”)

And then John can update “foo” ‘s API version 1 to include this API contract. He can do it by calling this REST API:

And then John can use Contract Server’s REST API to trigger contract testing against app “Foo” in his local environment (Port 8001). Once triggered, Contract Server would build mock requests according to the API contract and then send to the target API endpoint. The testing result and the whole Request/Response context information would be returned.

The Contract Test REST API would be like this:

Request:

The Response may look like this:

Next Story: Mary use the contract to kick start stub server

Thie time, Mary start a local Contract Server with port 8000. And then she create the App(John’s app “Foo”), Version and Contracts just like the last story. Again, it is done by REST API calls.

But at the last step, thie time she would not call the API contract testing operation ( /api/v1/apps/{{appId}}/wiretests ). Instead, she want to wire a stub server. Hence, she would call the API wire stub operation ( /api/v1/apps/{{appId}}/wirestubs ).

After that, a stub server which responses according to the API contracts is started at local port 8001.

And then, Mary can test the hello API with the stub server. She can send a request like this:

And she would get this response:

After Mary’s testings, Mary can shutdown the stub server by this API:

Or, she can shutdown the whole local Contract Server instead.

 


Credit to Spring-Cloud-Contract & WireMock

We have to give credit to Spring-Cloud-Contract & Wiremock because this project is inspired by them. We like Spring-Cloud-Contract’s design and usage of value(stub(...), test(...)) so we bring it in rest-in-contract. We also appreciate Wiremock which showing us an well-made product for case study/feature analysis to help us make our own new product.

關於mobile做authentication security

[itdog][mobile, security, oauth][其實IoT可能都關事(唔熟)]

關於mobile做authentication security問題。
之前可能講過, but我都再講多次, 重要野唔怕一講再講。

Mobile App裝o係Mobile,你就要預左好似frontend javascript咁赤裸俾人睇晒。
因為hacker可以crack mobile
=>拎到APK file
=>decompile做source code

我舉個例子,mobile app用OAuth Authorization Code Grant Flow做authentication。
好多人都係咁做,好多industrial專業engineer都可能係咁做。(就算vendor如pingIdentity, 佢地個網都係咁講架ja)

但以大眾做法,我(or hacker)好似上面咁講,decompile source code,拎到你hard code既client id同client secret。
拎到client id同secret,就有security位俾hacker利用。
玩法可能好多(e.g: phishing, 假client etc)。
實際security impact,視乎OAuth以至成個architecture﹑business flow擺位,以及token權力範圍。

其實就算做OAuth,都有較安全做法。
就係當mobile app install落部機時,立即dynamic registration去register一個per device既OAuth client。
reg完OAuth, 然後立刻將device client register到application的DB。
runtime時,device每次perform operation,都是以device名義去access server。
server side每次checking就要check token validity,同埋device authentication(對返DB record)。

consideration其實有幾個point不過skip費事太長氣。
淨係講最重要一點:

hacker就算crack左你個mobile app睇晒source code都冇用,因為每部device裝完機到runtime都會拎個dynamic client去,根本唔會有個hard code左既id/secret儲o係source code。

(其實玩 IoT 可能都係類似咁,起碼要有device registration, rather than only IoT app registration)