On Tue, Apr 28, 2015 at 12:46:14AM +0530, Ratnadeep Debnath wrote:

Hi all,

I want to propose a talk on our ongoing project: statscache[1] in Flock 2015. I am sharing a draft of the proposal for your review:

Title

Statscache, a near realtime intelligent caching layer empowering faster data representation

Summary

Fedmsg and fedmsg based tools: datanommer, datagrepper, etc. have indeed taken communication and coordination across various components in the Fedora ecosystem to an all new level. Currently, we are using the raw datagrepper APIs to render various widgets and data visualization. However, due to the massive amount of data stored in datagrepper backend, searches tend to be slower. Also, consuming the raw datagrepper API leads to increased data payload over the network and writing complex client side logic to format the data.

Statscache is an attempt to solve the above problems and also to create a generic framework to empower powerful data visualization. It achieves this by listening to fedmsg events and storing the denormalized and clean data in desired format for faster retrieval. It also aims to become an intelligent caching layer based on "Single source of truth" model and publishing REST and websocket API around it to enable faster and realtime data visualizations. This could become the backbone for empowering various widgets in Fedora hub.

Topology diagram: https://raw.githubusercontent.com/fedora-infra/statscache/develop/docs/diagr...

+1 from me :)