15:00:35 <regXboi> #startmeeting neutron_northbound
15:00:35 <odl_meetbot> Meeting started Fri Aug 14 15:00:35 2015 UTC.  The chair is regXboi. Information about MeetBot at http://ci.openstack.org/meetbot.html.
15:00:35 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:35 <odl_meetbot> The meeting name has been set to 'neutron_northbound'
15:00:37 <flaviof> #info flaviof
15:00:43 <regXboi> #chair flaviof, edwarnicke
15:00:43 <odl_meetbot> Current chairs: edwarnicke flaviof regXboi
15:00:52 <regXboi> #topic roll call and agenda bashing
15:00:54 <regXboi> #info regXboi
15:01:00 <edwarnicke> #info edwarnicke
15:01:10 <mestery> #info mestery in lurk mode
15:01:21 <regXboi> #link https://wiki.opendaylight.org/view/NeutronNorthbound:Meetings agenda in its usual place
15:01:38 <flaviof> regXboi: mestery: maybe we can also talk about stable/kilo for networking-odl issie
15:01:38 <regXboi> #info regXboi left the agenda light because we need more discussion time on itema
15:01:39 <flaviof> issue
15:02:03 <regXboi> flaviof: there is lots of time for that :) - it's all open mike today
15:02:09 <flaviof> ack
15:02:12 <yamahata> #info yamahata
15:02:16 <regXboi> #topic action items from last meeting
15:02:31 <regXboi> #info regXboi to consult with GBP folks about bug 3968 and why they are calling isXXX() methods directly
15:03:01 <regXboi> I actually did talk to alagalah yesterday and this defect isn't coming from GBP - so I'm now planning on rejecting it as theoretical
15:03:11 <regXboi> and not from a real code example
15:03:26 <regXboi> #info regXboi has consulted and is not convinced bug 3968 is real
15:03:36 <regXboi> #info  regXboi, edwarnicke, flaviof to find a document contact vict.... volunteer
15:03:40 <regXboi> still looking :)
15:03:49 <regXboi> step right up folks - your chance to help out :)
15:04:03 <regXboi> yeah - we'll carry that for another week...
15:04:10 <regXboi> #action  regXboi, edwarnicke, flaviof to find a document contact vict.... volunteer
15:04:24 <mestery> flaviof: Ack
15:04:24 <regXboi> #info regXboi to kick off threads on networking-ODL
15:04:33 <regXboi> yeah I was late on this, but they are out there:
15:04:43 <regXboi> #link https://lists.opendaylight.org/pipermail/neutron-dev/2015-August/000275.html plugin or ML2
15:04:53 <regXboi> #link https://lists.opendaylight.org/pipermail/neutron-dev/2015-August/000276.html where should it live
15:05:02 <regXboi> #link https://lists.opendaylight.org/pipermail/neutron-dev/2015-August/000277.html two projects or one
15:05:10 <regXboi> and that's the action item list
15:05:12 <regXboi> so....
15:05:21 <regXboi> #topic Beryllium/Open Mike
15:05:22 <mestery> nice work regXboi
15:05:35 <regXboi> flaviof: you have the floor :)
15:05:56 <regXboi> #info flaviof to talk about stable/kilo networking-odl issue
15:06:06 * regXboi nudges flaviof to wake up
15:06:08 <flaviof> regXboi: ack, thsnk.
15:06:37 <flaviof> #link https://lists.opendaylight.org/pipermail/neutron-dev/2015-August/000284.html stable/kilo issue
15:07:22 <flaviof> basically, when we created stable/kilo for networking/odl we introduced a mis-alignment in the versions of pbr
15:08:37 <mestery> flaviof: We need to get the proposal bot syncing accurate stable/kilo changes into networking-odl
15:08:38 <mestery> Is that right?
15:09:06 <flaviof> mestery: yeah. I do not know what it means, but yes, that sounds right :)
15:09:11 <mestery> :)
15:09:25 <edwarnicke> mestery: Side note: I think you just ticked up one more on the plus side for leaving networking-odl at OS ;)
15:09:37 <flaviof> edwarnicke: +1
15:10:40 <regXboi> #info problem is we need the proposal bot syncing accurate stable/kilo changes into networking-odl
15:10:49 <flaviof> that is all on this topic, really; just need to get done... maybe mestery or some competent o/s folk can sign up for that?
15:10:58 <regXboi> #info this is a +1 for leaving networking-odl at openstack
15:11:36 * flaviof is done... next topic?
15:12:11 <regXboi> well... I'd rather keep the other discussions to the ML threads
15:12:21 <regXboi> so I guess it is time to open this pandora's box
15:12:27 <regXboi> #info changing the wire protocol
15:12:40 <flaviof> mlemay: ping ^^
15:13:13 <regXboi> #link https://wiki.opendaylight.org/images/8/8d/Experiences_with_Neutron.pdf Dec's summit talk
15:13:56 <regXboi> so folks, I'm not really convinced this isn't going to cause more trouble than it's work
15:13:59 <regXboi> er worth
15:14:16 <edwarnicke> regXboi: I think there are two things here
15:14:23 <edwarnicke> regXboi: 1)  Figuring out HA/Scalability
15:14:33 <edwarnicke> 2)  The possibility that that might entail wire protocol changes
15:14:41 <edwarnicke> I think we *have* to do #1
15:14:46 <flaviof> how about we talk about each observation
15:14:48 <flaviof> ?
15:14:55 <regXboi> edwarnicke: do we want to step back and walk through all of the observations and how to deal with it?
15:14:57 <regXboi> flaviof: +1
15:14:58 <flaviof> otherwise it is too easy to get off topic.
15:15:00 <edwarnicke> I think #1 *might* imply #2... but could be convinced otherwise
15:15:23 <edwarnicke> flaviof: regXboi I'm fine with that if thats how you'd like to handle it :)
15:15:29 <regXboi> I think I do
15:15:38 <regXboi> try and make some progress in the 45 minutes
15:15:42 <flaviof> yeah, thanks edwarnicke !
15:16:21 <regXboi> #info current openstack ml2 odl driver is single threaded and blocks neutron server (post-commit) until ODL responds
15:16:34 <regXboi> um... mestery - do you know of an ml2 driver that is multi-threaded?
15:16:47 <regXboi> I can't think of one, and I'm not sure why I really care
15:16:49 <yamahata> Now ML2 folks are aware of the issue and thinking of asynchronous one
15:16:49 <flaviof> so... on that one; armax, mestery may know the intricate details that explain that?
15:16:51 <mestery> lol
15:16:54 <yamahata> there is PoC
15:17:12 <edwarnicke> regXboi: flaviof mestery Are plugins traditionally multi-threaded?
15:17:17 <yamahata> and they are also aware of synchronizing issue under discussion.
15:17:23 <armax> regXboi: well that’s not entirely accurate
15:17:24 <mestery> The driver itself needs a re-write at this point.
15:17:35 <mestery> The problem is due to how it handles re-sync, we had to make it single-threaded
15:17:36 <flaviof> mestery, yamahata, armax: is this something that would go away if odl used plugin instead of ml2?
15:17:39 <armax> neutron uses greenthreads and they are non-blobkcing
15:17:42 <mestery> And even worse: IT uses a local file lock
15:17:43 <regXboi> armax: which of my many statements is not accurate :)
15:17:50 <mestery> armax: ++
15:18:06 <armax> the fact that the driver is single threaded blocking
15:18:08 <armax> regXboi: ^
15:18:24 <yamahata> Even with monolithic plugin, the same problem needs to be solved.
15:18:25 <regXboi> armax: I'm quoting the observation
15:18:35 <armax> regXboi: and I am not blaming you
15:19:09 <regXboi> help me out here - is this just about scaling the management plane?
15:19:10 <yapeng> sorry if it is a stupid question, why need a multi-thread driver? most of ML2 is configuring network/subnets/...
15:19:15 <edwarnicke> armax: mestery How does neutron/OS multi-threadedness work around the GIL?
15:19:28 <flaviof> yamahata: ack, thanks. my rush to move away from ml2 is that this is a 'greener grass on the other side illusion'. That info helps.
15:19:45 * edwarnicke admits its been a while, but still remembers multi-threadedness in python being radically hamstrung by the GIL...
15:20:13 <armax> greenthreads are different from regular OS threads
15:21:06 <regXboi> I'm not sure at this point that we need to do anything other than include this in what we were already contemplating in the rewrite
15:21:15 <regXboi> is that a fair conclusion??? ^^^^^^
15:21:23 <armax> regXboi: +
15:21:55 <edwarnicke> armax: Understood about greenthreads... and looking deeper it seems that the GIL isn't so relavent to them, because there is never more than a single greenthread running at once... did I get that right?
15:22:04 <armax> yes
15:22:08 <mestery> regXboi: ++
15:22:16 <regXboi> I'm going to write in an agree here folks :)
15:22:25 <mestery> regXboi: Please do before we go off the rails
15:22:32 <flaviof> regXboi ++
15:22:37 <armax> edwarnicke: thread execturion can still interleave though and that’s where teh interference comes into place
15:22:46 <regXboi> #agree this observation should be bundled as a consideration/requirement for the rewrite ML discussion
15:22:50 <viveknarasimhan> regXboi +
15:23:05 <armax> regXboi: ideally we’d know what we’re rewriting into
15:23:12 <armax> regXboi: we don’t want to swap mess with mess
15:23:14 <regXboi> #info second observation: db synchronization issue
15:23:15 <edwarnicke> armax: Yep... if you are waiting on IO, you should yield so the next guy gets  chance :)
15:23:22 <armax> regXboi: but I guess that I am stating the obvious
15:23:52 <edwarnicke> armax: My experience has been that the best way to not swap mess with mess is small incremental steps of rewrite with continuous measuring of the resulting desired improvement
15:23:56 <regXboi> #info regXboi thinks this is a bit of a bogus observation (at least as presented in the slide)
15:24:06 <flaviof> armax: no you are not, buddy. I think it is a very valid point.
15:24:34 <regXboi> armax: yes - I'm saying we need that to be part of that discussion or we may a boot
15:24:41 <regXboi> er boat (or whatever)
15:24:42 <edwarnicke> regXboi: Say more... because I see it as being a big deal, and would like to understand why you don't
15:24:43 <edwarnicke> ?
15:25:04 <regXboi> so... when I look at this slide - I see multiple independent neutrons
15:25:14 <regXboi> therefore, they *don't* know about each other
15:25:27 <regXboi> and so I can't do the steps that come to the ODL controller
15:25:42 <regXboi> unless (and here I give my evil smile)
15:26:01 <regXboi> you are implying that UUIDs from the upstream neutrons can collide (gotcha :) )
15:26:28 <flaviof> regXboi: not sure if they are truly " independent neutrons"
15:26:41 <edwarnicke> regXboi: I think he is talking about different control nodes for the same logical neutron
15:26:48 <edwarnicke> regXboi: But sharing the same underlying DB
15:26:53 <regXboi> edwarncike, flaviof: unfortunately, they are drawn that way
15:26:53 <viveknarasimhan> flaviof: they are independent threads
15:26:57 <flaviof> regXboi: can't they be O/S HA ?
15:27:17 <regXboi> edwarnicke, flaviof: I consider this a poor diagram because it leaves that to the reader
15:27:20 <viveknarasimhan> they are not aware of each other, but they work on the state from a persistent store (like DB) etc
15:27:35 <flaviof> Is Wojciech here?
15:27:41 <regXboi> don't see him
15:27:46 <flaviof> k
15:27:49 <regXboi> I'm willing to table this one for a week and get an update
15:27:59 <regXboi> I suspect you are correct, but I'm standing on my interpretation for now :)
15:28:05 <yamahata> DB is sahred. e.g. UUID uniqueness is guaranteed by db unique contstraint.
15:28:38 <regXboi> #agreed tabling this observation for a week to get clarity of assumptions
15:28:42 <regXboi> er
15:28:43 <regXboi> #unto
15:28:45 <regXboi> #undo
15:28:45 <odl_meetbot> Removing item from minutes: <MeetBot.ircmeeting.items.Agreed object at 0x1a8c1d0>
15:28:54 <regXboi> #agreed tabling this observation for a week to get clarity of assumptions
15:28:58 <regXboi> ok, so that did work
15:29:07 <regXboi> yamahata: understood - I just want Dec to confirm that
15:29:36 <flaviof> another question I have is on the fact that these slides were done using icehouse. that predates networking-odl, so keep that in mind....
15:29:44 <regXboi> #info observation: current ML2 ODL driver sync mechanism only runs when triggered by a new Neutron event
15:30:07 <regXboi> so, this one looks like more rewrite requirements - doesn't it?
15:30:51 <regXboi> although.... I also admit that the more I look at it, the more I don't get it
15:30:58 <yapeng> regXboi, this youtube link of Wojciech's presentation: https://www.youtube.com/watch?v=CL70MNgFeQs
15:31:24 <regXboi> #link https://www.youtube.com/watch?v=CL70MNgFeQs Dec's presentation on youtube
15:31:26 <regXboi> yapeng: thx
15:31:28 <flaviof> regXboi: agree; this is related to Neutron <-- ODL events, which are pretty much non-existent in current implementaion
15:31:40 <vthapar> regXboi: when ML2 driver goes out of sync it sets out_of_sync flag, but it is not able to trigger resync till next event comes in.
15:31:40 <flaviof> yapeng: thanks,
15:31:48 <edwarnicke> regXboi: I think #3 is basically that each control node tries for each event
15:31:55 <edwarnicke> So if something goes wrong, you are getting repeat failures
15:32:10 <edwarnicke> But I think it also opens us up to bad out of order issues
15:32:15 <regXboi> well... I'll still admit - I don't get this at all
15:32:30 <regXboi> because for this to even happen, things have to go horribly wrong in the first place
15:33:05 <yamahata> Right. basically db transaction then sending request to ODL.
15:33:07 <regXboi> and so I'd rather fix what causes things to go horribly wrong
15:33:10 <flaviof> I'm stuck too... there is no T3 ;)
15:33:28 <regXboi> and then come back to this
15:33:52 <yamahata> the order of db transaction and the order of request to ODL can be reordered.
15:33:54 <regXboi> btw edwarnicke: the "things go horribly wrong" is very close/overlays the HA
15:33:56 <yamahata> even with single neutron server.
15:34:03 <edwarnicke> regXboi: I don't quite see it that way (which may not mean that I'm wrong)...
15:34:20 <vthapar> I've seen out_of_sync happen with single neutron server also
15:34:36 <edwarnicke> regXboi: I see it as sort of this issue.  A db event happens, you have n control nodes.  You get n calls to ODL.  Nothing guarentees they are in the same order, and it means n times the load
15:34:49 <edwarnicke> vthapar: Please say more
15:34:52 <yamahata> vthapar: Cool. I haven't been able to see it with single neutron.
15:34:55 <regXboi> yamahata: if neutron is reordering events so that the events to the db are different that the events southbound, that's a horrible mistake
15:34:58 <flaviof> yamahata: ic, so it is 'okay' to expect subnet create before net crate, as an example?
15:35:20 <regXboi> no it shouldn't be
15:35:23 <regXboi> I'm sorry
15:35:26 <vthapar> seen the issue when spawning VMs with multiple NICs and then clearing them out. I'll hve to get more details from our test team.
15:35:47 <regXboi> vthapar: that's a bit different then subnet coming before network
15:35:52 <yamahata> flaviof: In theory, yes. But I think it should be cared by neutron side somehow.
15:35:56 <regXboi> so let's be careful about reording
15:36:03 <regXboi> er reordering
15:36:04 <yamahata> e.g. backgroundly syncing.
15:36:05 <viveknarasimhan> vthapar: when you say single neutron server, what was your api-worker configuration
15:36:31 <regXboi> folks: I'm going to budge until 45 past the hour for this - we have more observations to cover
15:36:37 <viveknarasimhan> vthapar : even a single neutron-server can be run with n-workers which would enable concurrent handling of REST requests
15:36:44 <regXboi> budget that is - unless we come to a conclusion sooner :)
15:36:48 <vthapar> viveknarasimhan: I believe it was default in my setup, where I saw it not so frequently. can get details from test folks as they used to run into it frequently enough.
15:36:50 <yamahata> regXboi: In ideal world, yes. But the difficult cames from the fact that we have two state to be synced.
15:36:56 <viveknarasimhan> vthapar: better to try with single neutron -server single api-worker and see if issue reproduces
15:37:05 <flaviof> yamahata: ack. I think that explains all the changes in master regXboi has been doing, so nn code makes no sanity checks on things like that (i.e. net being created before subnet).
15:37:19 <regXboi> yamahata: I'm sorry, but that answer doesn't quite hold water
15:37:25 <yamahata> And ML2 team is aware of this issue.
15:37:41 <regXboi> telling the DB order A and telling southbound order B is *evil*
15:38:06 <yamahata> has been discussed on the topic. there are (unfinished?) PoC to some extent
15:38:23 <regXboi> actually, I'd like to table this until we see what comes out of the ML2 team
15:38:30 <regXboi> because that's going to be what we have to work with
15:38:42 <regXboi> unless somebody wants to volunteer to track it :)
15:38:54 <flaviof> sorry if I sound like a broken record folks: is this a ml2 only issue? is this something a plugin would have to 'worry about'?
15:39:13 <regXboi> flaviof: IIRC, the plugin takes this responsibility on itself
15:39:26 <regXboi> flaviof: so reordering of southbound requests is up to the plugin
15:39:27 <flaviof> regXboi: ack. thanks.
15:39:49 <yamahata> Yes, syncing status is common problem among controllers.
15:39:57 <regXboi> flaviof: in my mind this is a +1 argument for plugin, pending where the ML2 team comes out
15:40:08 * flaviof recalls quote from spiderman: more power, more responsabilities
15:40:34 <regXboi> can we agree on tabling this/making it part of the rewrite?
15:40:44 <yamahata> even with monolithic plugin, there remains same problem.
15:40:47 <flaviof> regXboi: ++
15:40:55 <edwarnicke> yamahata: regXboi I guess my more simple questions would be these:
15:41:03 <flaviof> regXboi: but I have a feeling we are moving problems and solving none....
15:41:04 <edwarnicke> Can events arrive out of order at an ML2 driver?
15:41:12 <edwarnicke> Can events arrive out of order at a plugin?
15:41:31 <regXboi> edwarnicke: define events - db events? some other events?
15:41:32 <yamahata> One extreme solution by opencontrail is not to use neutron db. always using controller side.
15:41:41 <edwarnicke> It *sounds* like they can arrive out of order at an ML2 driver... but I couldn't tell if that was true of a plugin
15:41:58 <edwarnicke> yamahata: Interesting, please say more
15:42:01 <regXboi> afaik, db events do not arrive out of order to an ML2 driver
15:42:27 <regXboi> uh... I'm really not interested in following the opencontrail route
15:42:38 <regXboi> I mean really really really
15:42:47 <regXboi> as in -2 really
15:43:07 <edwarnicke> regXboi: If a change comes in to the db (for many things we are listening for db changes) are we gauranteed to hear it in the order it was applied?
15:43:14 <yamahata> opencontrail plugin just passes through all the request to controller. It means they re-implement full neutron functionality in the controller.
15:43:18 <flaviof> regXboi: while I'm not interested in contrial; I think understanding a model that is proven to work well may be helpful
15:43:26 <flaviof> at least if we are re-writting
15:43:33 <flaviof> which we are, right?
15:43:42 <yamahata> there is no db operation (or no meaningfull operation) in neutron side.
15:43:43 <edwarnicke> regXboi: I'm not necessarily disagreeing with your conclusion... but I am keeping a more open mind in the interum... sometimes an ugly solution is better than none at all
15:43:44 <regXboi> flaviof: note what yamahata said
15:43:53 <regXboi> "re-implement full neutron functionality"
15:43:58 <vthapar> just checked some older mails from test team, exceptions in rest calls to ODL can cause sync issues.
15:43:59 <regXboi> that's a BIG statement
15:44:21 <regXboi> edwarnicke: I have the scars from this - I'm not interested
15:44:23 <vthapar> any updates/deletes are "lost" on resync and cause further problems.
15:44:50 <regXboi> folks - we are coming up on the budgeted time
15:45:04 <regXboi> do we want to keep going and leave the other observations for next week?
15:45:14 <regXboi> #info lots of discussion about observation 3 - see the logs
15:45:23 <edwarnicke> regXboi: Other than taking responsibility for storing the data (which we have to do anyway), what additional things does yamahata 's suggestion imply we need to do?
15:45:38 <flaviof> vthapar: ack. that is inline with Observation#3 in Dec slides, right?
15:46:00 <regXboi> edwarnicke: all of the checking/tracking/state management that I've been taking out
15:46:10 <edwarnicke> regXboi: OK
15:46:11 <vthapar> falviof: yes, though I'd say #2 and #3 both.
15:46:15 <yamahata> Another possibility is to sync intelligently. e.g. adding sequencenumber to each request. and sync backgroundly
15:46:31 <flaviof> vthapar: ack.
15:46:33 <edwarnicke> yamahata: Please say more :)  I've heard this suggestion before as well :)
15:46:37 <regXboi> honestly, vthapar's comments say the current sync mechanism needs rewriting
15:46:59 <yamahata> sequence number can be added to each status.
15:47:05 <vthapar> yamahata: one more gap in resync logic is it doesn't track failed events and way it does only adds are taken care of.
15:47:31 <yamahata> so that we can detect reorder or event loss.
15:47:37 <vthapar> any deletes that fail stay in ODL but gone from Neutron DB.
15:48:06 <regXboi> whoa... vthapar
15:48:07 <yamahata> tracking how behind ODL is from neutron.
15:48:56 <regXboi> actually - I'd argue that the ML2 connection to ODL being post-commit is a bit ... well ... broken
15:49:07 <yamahata> regXboi: agree.
15:49:17 <regXboi> but that's also for the rewrite to consider
15:49:25 <flaviof> regXboi: ++
15:49:31 <vthapar> regXoi: ++
15:49:32 <regXboi> I'd argue that we ought to take a page out of our own book
15:49:37 <vthapar> regXboi: ++
15:49:39 <regXboi> and go the pre-commit/post-commit route
15:49:49 <flaviof> regXboi: when we say 'rewrite', can we be a little more specific?
15:49:49 <regXboi> I*Aware anyone ????
15:50:09 <regXboi> flaviof: I'm saying that currently we register for post-commit events
15:50:18 <flaviof> rewrite could be: fix sync .... to use plugin...
15:50:19 <regXboi> I think we have to register for pre-commit events as well
15:50:29 <regXboi> and use the pre-commit event to veto a change
15:50:36 <regXboi> and the post-commit event to make the change
15:50:37 <flaviof> regXboi: ack. make sense.
15:50:46 <regXboi> like I said - I*aware again
15:51:08 <regXboi> have we run the gamut on this one for now?
15:51:19 <regXboi> I'm not saying we've closed it
15:51:32 <flaviof> with I*aware going away, we are taking a step in the wrong direction? is that what you mean regXboi ?
15:51:44 <regXboi> flaviof: no - in ODL it should go away
15:51:59 <regXboi> between OS and ODL, the pattern makes more sense
15:52:11 <regXboi> that's what I'm saying
15:52:27 <flaviof> regXboi: ok. thanks.
15:52:47 <regXboi> #info observation 4: no switch type independent configuration for mapping of neutron physical networks to physical interfaces
15:52:57 <regXboi> edwarnicke: remind me again why we care about this one?
15:53:10 <regXboi> is this the responsibility of the providers?
15:53:15 <regXboi> er isn't this ....
15:53:59 <edwarnicke> regXboi: You can make a case for that being the providers responsibility
15:54:13 <edwarnicke> regXboi: But the net-net is its a general problem that everybody has to have some kind of hack around
15:54:35 <flaviof> regXboi: I think this is Dec wishing that networking-odl / neutron could be made to help more by giving hints on the mapping of neutron-port to OVS ports
15:55:03 <regXboi> edwarnicke, flaviof: both of those imply NN knowing more about the physical topology that I'm personally comfortable with
15:55:28 * regXboi doesn't want to get into the inventory business - other projects are doing that
15:55:32 <flaviof> it still needs to live in a world where there is no OVS, or no OVS ports for every neutron port object.
15:56:30 <flaviof> regXboi: understtod. at some point, someting needs to map neutron port to an object that makes sense to the netvirt.
15:56:57 <regXboi> flaviof: yes, but I'm not yet convinced that is *us*
15:57:15 <edwarnicke> flaviof: I think part of this is a deeper neutron thing that seems to be (right or wrong) vexing a bunch of folks...
15:57:22 <flaviof> whether that mapping lives in nn or another project is the real question. Sounds like nn is not it then. ;)
15:57:39 <regXboi> flaviof: I said, I"m not yet convinced
15:58:12 <edwarnicke> flaviof: I think the basic argument comes down to the fact that the mapping is going to be communicated different ways by different underlying infra
15:58:18 <regXboi> edwarncike - can you start a ML thread on this one?
15:58:32 <regXboi> since (IIRC) you have a deep understanding of this
15:58:54 <edwarnicke> flaviof: That said... having a common place to stash the info has some appeal... I'm really not decided in my own mind just yet
15:59:06 <edwarnicke> regXboi: Not deep, no
15:59:10 <regXboi> #action edwarnicke to launch neutron-dev ML thread on observation #4 from Dec's talk
15:59:21 <regXboi> edwarnicke: deeper than mine
15:59:25 <edwarnicke> regXboi: But it might be good to solicit a deeper conversation on neutron-dev from Woj
15:59:38 <regXboi> edwarnicke: ++ that's the reason for the ML thread
15:59:42 <regXboi> so that Woj can chime in
16:00:02 <regXboi> ok folks we'll table the others for next week
16:00:09 <regXboi> #topic cookies
16:00:12 <regXboi> #undo
16:00:12 <odl_meetbot> Removing item from minutes: <MeetBot.ircmeeting.items.Topic object at 0x1ce0050>
16:00:25 <regXboi> #action regXboi to update ML thread on rewrite with notes from this meeting
16:00:28 <regXboi> #topic cookies
16:00:31 <regXboi> #endmeeting