Distributed Node with Distributed Quota System (DNDQS)

最近因為某些原因而去研究一下如何在distributed的系統上做API quota機制。

首先是看了一篇研究論文 http://www.ssrc.ucsc.edu/Papers/pollack-msst07.pdf [Quota enforcement for high-performance distributed storage systems]
那篇論文裡所design的系統是用centralized quota server去做control發放voucher給nodes,nodes則能利用那些voucher去行使quota。
這倚賴了centralized quota server做sync quota的處理機制。
而其避免performance bottleneck的做法是一直用slide window protocol的方式預留了一些voucher在nodes那邊。

 

後來我自己也借鑒了其一部份想法design了另一個solution,我稱為Distributed Node with Distributed Quota System (DNDQS)。

Concept Powerpoint PDF:

Distributed Node with Distributed Quota System (PDF)

Introduction:

基本想法是把quota的概念由quota server centralized control的data變成node level distributed而流動的data。
然後透過consistent hashing把nodes固定生成一個virtual circular node chain。
然後每一個node都按需要而沿node chain向下一個node發送request,請求下一個node把它的quota填補自身node的quota。
每一個node都會進行以上logic,所以unbalanced distributed quota會drive quota propagation從而逐漸balance。

Characteristic:

  • Distributed quota on nodes.
  • Make use of consistent hashing to form an ordered circular chain of nodes.
  • Balance node quota by mono-directional flow of quota through the node chain.

Benefits:

  • No need to sync quota to Quota Server.
  • Distributed Quota would eventually be balanced.

這個solution的design只是一個初步想法,我未有實際POC過,實際應用可行性及數據有待考證。
但有一個很大的潛在問題,就是當要support到有大量user各自的quota的時候,distributed quota data就會變得很擁腫,overhead會很大。

Leave a Reply

Your email address will not be published.