17:02:06 <colindixon> #startmeeting md-sal interest
17:02:06 <odl_meetbot> Meeting started Tue Dec  9 17:02:06 2014 UTC.  The chair is colindixon. Information about MeetBot at http://ci.openstack.org/meetbot.html.
17:02:06 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:02:06 <odl_meetbot> The meeting name has been set to 'md_sal_interest'
17:02:12 <colindixon> #topic agenda bashing
17:03:34 <colindixon> #link https://wiki.opendaylight.org/view/MD-SAL_Weekly_Call#Agenda the agenda
17:03:50 <colindixon> #info Mouli will share some performance evaluation results and solicit feedback from the community.
17:06:09 <colindixon> #chair abhijitkumbhare
17:06:09 <odl_meetbot> Current chairs: abhijitkumbhare colindixon
17:06:29 <colindixon> #topic Helium MD-SAL Data Store & OF Plugin Performance Analysis
17:06:55 <colindixon> #info slides will be posted after the meeting (there is a webex recording)
17:08:41 <colindixon> #info the goal is to characterize the performance of the MD-SAL DOM Data Store and OF plugin (as well as scalability and reliability)
17:09:28 <colindixon> #info used a switch simulator to measure flow rate (flows/second) and flow scaling (max flows) with a ~20 switches
17:09:58 <colindixon> #info used a Dell server with 64GB RAM, 8 cores and NWSim
17:10:52 <colindixon> #info used modified OpenFlow Plugin test provider (multi-threaded, batching flows, etc.)
17:13:04 <colindixon> #info tested DOM datastore alone (no OFplugin), DOM datastore alone (no OFplugin, no notifications), OF drop test (skip DOM datastore), DOM datatstore + OF plugin, and DOM datastore (no notifications) + OF Plugin
17:13:58 <colindixon> #info this was informed by finding bottlenecks, e.g., they found that notifications were expensive
17:15:15 <colindixon> #topic observations so far
17:15:42 <colindixon> #info originally were seeing <100 flows/second from the MD-SAL data store + OpenFlow plugin
17:16:10 <colindixon> #info potential bottlenecks here
17:16:16 <colindixon> #undo
17:16:16 <odl_meetbot> Removing item from minutes: <MeetBot.ircmeeting.items.Info object at 0x247be90>
17:16:18 <colindixon> #info potential bottlenecks here:
17:16:56 <colindixon> #info * data change notifications on WriteTxCommit processing
17:17:10 <colindixon> #Info * single-threaded commit processing compounds this problem
17:17:29 <colindixon> #info * the merge operation processing overheads were much higher than for put
17:20:01 <colindixon> #info removing the use of data change notifications in the process resulted in ~5000 flows/second
17:20:45 <colindixon> #info notifications are two operations (first creating listener trees and then difference computation), the overhead seems to be in the second
17:21:48 <colindixon> #info this ~5000 flows/second was with the OFplugin, but with no clustering (at least not yet)
17:24:48 <colindixon> #info they do note that the existing MD-SAL microbenchmark on a laptop-grade system hit 10s of thousands of operations per second
17:25:44 <colindixon> #info there were three key differences: (1) using normalized nodes instead of binding aware interfaces, (2) using notifications, (3) using a very simple model instead of the more complex flows
17:26:08 <colindixon> #info rovarga seems to think that (1) would cause a lot of overhead
17:29:06 <colindixon> #topic possible fixes
17:30:10 <colindixon> #info tony says one issue could be that there is a translation from MD-SAL internal data change events to the one defined in the API
17:34:51 <colindixon> #info rovarga thinks the right approach is to migrate the difference computation to the client, not the data store
17:37:37 <colindixon> #info the logic is that this would allow the client to do just the computation it needs, not all the way down to the leaves
17:40:32 <colindixon> #info rovarga argues that getting the subscription API implemented in way that allows for this kind of optimization would be difficult
17:41:05 <colindixon> #info colindixon says he thought that the triggering scope did this, e.g., subtree vs. current node vs. current node + children
17:41:35 <colindixon> #info rovarga says that this is *just* triggering scope, not the scope of the changes that are provided
17:42:28 <rovarga> #info the problem is that in order to know whether the scope has triggered, we need to perform full comparison. there is no way to say 'this is granular enough'
17:44:40 <colindixon> rovarga: thanks
17:46:15 <rovarga> #info and once we know it triggered, we need to also calculate all the nodes which changed, as we do not know what the user will ask for
17:47:12 <rovarga> #info a navigable tree of what has changed would solve this -- the app can navigate, find it out, and ask for DTOs which it is interested in
17:51:02 <colindixon> #info Uyen says that the current apps seem to be mostly based on data changes, and it appears (from this discussion) that this might not scale well
17:51:37 <colindixon> #info given that Uyen asks what the performance guidelines for using the MD-SAL are
17:52:59 <colindixon> #info rovarga and muthu answer that “it depends on your application”
17:53:38 <colindixon> #info colindixon restates the question “given that you want to have complex models with high update rates, is the answer don’t use data change notifications?”
17:54:37 <colindixon> #info rovarga says yes, but there may be APIs that allow for better scoped notifications and thus less pain here
17:55:18 <rovarga> #info in general: batch as much as possible, listen to the minimal set you need, perform put() [which pushes you to single-writer-per-subtree]
17:57:51 <rovarga> #info try to match what the produces put() and what the consumers trigger on
17:57:51 <colindixon> #topic wrap-up
17:58:15 <colindixon> #action muthu will send out data (profiler and performance numbers)
17:58:41 <colindixon> #action Mouli will send mail about possible missing/dangling flows
17:58:55 <colindixon> #action Muthu/Mouli to post slides to wiki
18:00:36 <colindixon> #action the community needs to understand the right patterns to use the MD-SAL to get decent performance
18:02:00 <colindixon> #endmeeting