15:01:19 <regXboi> #startmeeting neutron_northbound
15:01:19 <odl_meetbot> Meeting started Fri Sep 11 15:01:19 2015 UTC.  The chair is regXboi. Information about MeetBot at http://ci.openstack.org/meetbot.html.
15:01:19 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:19 <odl_meetbot> The meeting name has been set to 'neutron_northbound'
15:01:27 <regXboi> #topic agenda bash and roll call
15:01:31 <regXboi> #info regXboi
15:01:42 <yamahata> #info yamahata
15:01:56 <regXboi> #info apologies for no meeting last week, there were a dirth of committers
15:02:26 <regXboi> #link https://wiki.opendaylight.org/view/NeutronNorthbound:Meetings agenda from last week, which we'll start with
15:02:43 <edwarnicke> #info edwarnicke
15:02:49 <regXboi> and we have a quorum
15:03:16 <regXboi> #topic action items
15:03:31 <regXboi> item: edwarnicke, flaviof, regXboi to look at bug 3990
15:03:47 <regXboi> I haven't looked at this, so I think we'll have to carry it as still open, what say you edwarnicke?
15:03:51 <regXboi> #chair edwarnicke
15:03:51 <odl_meetbot> Current chairs: edwarnicke regXboi
15:04:38 <edwarnicke> regXboi: I haven't looked to closely at this either... but what's curious about it is that we use openjdk in the Jenkins Jobs
15:04:54 <regXboi> yeah, that's why I was a bit surprised
15:05:33 <regXboi> ok, let's put it back on the AI list
15:05:36 <regXboi> #action edwarnicke, flaviof, regXboi to look at bug 3990
15:05:46 <regXboi> item:  yamahata to check and comment on bug 4026
15:06:17 <edwarnicke> regXboi: If its a race condition we could just be getting lucky in the Jenkins Builds
15:06:20 <yamahata> I haven't. will look into it.
15:06:30 <regXboi> ok, back to AI for that
15:06:30 <yamahata> re 4026
15:06:37 <regXboi> edwarnicke: ack, yamahata: thx
15:06:45 <regXboi> #action  yamahata to check and comment on bug 4026
15:06:52 <regXboi> item regXboi to add IT test case for bug 4027
15:06:58 <regXboi> that patch has merged, so that item is done :)
15:07:07 <regXboi> item: yamahata to look at bug 4043
15:07:33 <yamahata> re 4043, the cause is identified, and the patches were uploaded.
15:07:40 <edwarnicke> yamahata: link?
15:07:53 * yamahata looking for it
15:07:55 <regXboi> #link https://git.opendaylight.org/gerrit/#/c/26266/ PS for 4043
15:08:12 <yamahata> thanks
15:08:16 <regXboi> although honestly, we need to fixup ALL of the places where we store CIDR blocks
15:08:43 <yamahata> and also 26267, I found the similar issue.
15:08:51 <regXboi> so... I'm really uncomfortable merging those patch sets
15:09:12 <yamahata> regXboi: Are you against the patch? or merge conflict issue?
15:09:16 <edwarnicke> https://git.opendaylight.org/gerrit/#/c/26266/ looks good to me, any idea why flaviof didn't merge?
15:09:18 <regXboi> neither of them have test cases so that jenkins can actually verify the change
15:09:27 <regXboi> edwarnicke: ^^^^^^
15:09:46 <regXboi> i.e. we *think* it is correct, but how do we *know* it is correct
15:09:49 <yamahata> Ah, sure. I'll add test cases.
15:09:54 * edwarnicke is less of a unit test zealot... but then regXboi is the one who bled to get our testing to where its at
15:10:08 <regXboi> regXboi has watched too many regressions happen
15:10:14 <regXboi> and I actually don't think this is UT
15:10:17 <regXboi> I think it is IT
15:10:18 <regXboi> :)
15:10:20 <edwarnicke> There are those who maintain I am a regression ;)
15:10:28 * regXboi isn't going to touch that one
15:10:54 <regXboi> yamahata: please add the test cases to each and ping me when you have them done and I'll review and merge them in
15:11:00 <regXboi> in the meantime
15:11:06 <yamahata> regXboi: will do
15:11:16 <regXboi> #action yamahata to add test cases to PS 26266 and 26267
15:11:29 <regXboi> #action regXboi to review and merge 26266 and 26267 once test cases are in place
15:12:07 <regXboi> speaking of which - we have a bunch of things to merge to prep tht
15:13:43 <regXboi> next item: yamahata to fix up https://git.opendaylight.org/gerrit/#/c/25500/
15:14:03 <regXboi> yamahata: actually, I have a lot of PS in flight that have that
15:14:14 <yamahata> I didn't have time to fix 25500.
15:14:27 <regXboi> yamahata: no worries, we'll carry it as an AI
15:14:46 <regXboi> #action yamahata to fix up https://git.opendaylight.org/gerrit/#/c/25500/
15:14:50 <regXboi> simply because
15:14:58 <regXboi> item regXboi to look at adding out of order IT tests to catch other issues like bug 4188
15:15:03 <regXboi> I haven't had time to do that either
15:15:04 <regXboi> so
15:15:07 <regXboi> #action regXboi to look at adding out of order IT tests to catch other issues like bug 4188
15:15:17 <regXboi> and last: everybody to think about how to deal with issues like https://bugs.opendaylight.org/show_bug.cgi?id=4188
15:15:27 <regXboi> I think that one needs to be part of the later discussion
15:15:57 <regXboi> under the heading of open patch sets
15:16:13 <regXboi> there are a bunch related to pulling out the CHMs
15:16:40 <regXboi> yamahata-san: I saw your -1 and answered - I really would rather clean up all the patterns in a single PS after everything merges
15:17:03 <yamahata> regXboi: I see, that's fine with me.
15:17:11 <regXboi> anything else or next topic?
15:17:23 <regXboi> #topic Berylliym
15:17:31 <regXboi> just an update here:
15:17:42 <regXboi> we have dependencies from the following downstream groups (so far): VTN, OVSDB, BGPVPN, LispFlowMapping, GBP
15:18:10 <regXboi> #info we have dependencies from the following downstream groups (so far): VTN, OVSDB, BGPVPN, LispFlowMapping, GBP
15:18:28 <regXboi> #info BGPVPN and GBP need to configure their jjb correctly so that the dependency appears in our merge
15:18:47 <regXboi> edwarnicke: can I task you with helping the bgpvpn and gbp folks get that done?
15:19:39 <regXboi> edwarnicke: ping?
15:21:53 <regXboi> ok, I'll take silence as a yes :)
15:22:28 <regXboi> #action edwarnicke to work with BGPVPN and GBP projects about getting their dependency on NN into their JJB configs so that integration on merge is handled correctly
15:22:47 <regXboi> #topic Other patch sets
15:23:10 <regXboi> I'm not going to put the ConcurrentHashMaps in here because that is (in my mind at least) just cleaning up
15:23:58 <odp-gerritbot> A change was merged to neutron: Migration to use MD-SAL Project  https://git.opendaylight.org/gerrit/26353
15:25:36 <regXboi> #link https://git.opendaylight.org/gerrit/#/c/26128/ northbound: return 404 when deleting unexisting uuid
15:26:04 <regXboi> if we can get the merge conflict fixed, I can see that merging
15:26:25 <yamahata> I'll rebase them.
15:26:39 <regXboi> yamahata: you might want to wait until the CHMs all land
15:26:47 <regXboi> some of them do a bit of changing to the NN
15:27:10 <yamahata> regXboi: okay. I'll wait. Please let me know when done.
15:27:18 <regXboi> yamahata: ack
15:27:46 <regXboi> #link https://git.opendaylight.org/gerrit/#/c/26127/  northbound: introduce a base class
15:27:58 <edwarnicke> regXboi: Apologies... was briefly distracted, the bgpvpn and gbp guys need their jjbs to depend on neutron?  Correct?
15:28:06 <regXboi> edwarnicke: yes
15:28:09 <regXboi> you have an AI on that
15:28:25 <regXboi> and it look slike 26127 needs a rebase as well
15:29:29 <regXboi> #link https://git.opendaylight.org/gerrit/#/c/25332/ BUG #4143 : Transation failed with OptimisticLockFailedException
15:29:43 <regXboi> edwarnicke: any reason we can't merge ^^^^^ ?
15:29:54 <regXboi> oh
15:29:55 <regXboi> I see
15:29:59 <regXboi> transaction chains
15:30:07 <yamahata> If transaction chains is necessary, I'll work on it.
15:30:14 <yamahata> Should I convert it into chain?
15:30:21 <regXboi> yamahata: please and thank you
15:30:26 <yamahata> got it.
15:30:32 <edwarnicke> yamahata: I can advise on transaction chains :)
15:30:35 <regXboi> oh heck
15:30:39 <edwarnicke> yamahata: I've done them before :)
15:30:43 <edwarnicke> yamahata: Just ping me on IRC :)
15:30:47 <yamahata> edwarnicke: thanks.
15:30:47 <regXboi> edwarnicke: let's get the CHM patches all merged
15:31:15 <regXboi> there is a whole series of them
15:31:18 <regXboi> https://git.opendaylight.org/gerrit/#/c/26551/
15:31:26 <regXboi> https://git.opendaylight.org/gerrit/#/c/26554/
15:31:36 <regXboi> https://git.opendaylight.org/gerrit/#/c/26577/
15:31:43 <regXboi> https://git.opendaylight.org/gerrit/#/c/26581/
15:31:53 <regXboi> https://git.opendaylight.org/gerrit/#/c/26578/
15:32:05 <yamahata> I'm catching up to review them. Not finished yet.
15:32:27 <regXboi> unfortunately, they get bigger as you go because they needed more changes to make work
15:32:28 <regXboi> :(
15:32:40 <regXboi> hey colindixon - you around?
15:32:42 <edwarnicke> regXboi: GBP jjb neutron dependency: https://git.opendaylight.org/gerrit/#/c/26855/
15:32:53 <colindixon> regXboi: mostly
15:33:00 <edwarnicke> regXboi: vpnservice jjb neutron dependency: https://git.opendaylight.org/gerrit/#/c/26856/
15:33:10 <regXboi> if so... those patches are why stable/li backport of yang is messy
15:33:38 <regXboi> edwarnicke: ack and I've +1'd both of them
15:33:56 <edwarnicke> yamahata: Are you cool with regXboi's response here: https://git.opendaylight.org/gerrit/#/c/26551/
15:34:06 <edwarnicke> yamahata: and here: https://git.opendaylight.org/gerrit/#/c/26554/
15:35:37 <regXboi> edwarnicke: scroll back to around 10:16 of this meeting for the answer
15:35:48 <regXboi> er 10:16 Central Time :)
15:35:59 <edwarnicke> yamahata: Could you un -1 then those two patches?
15:36:11 <yamahata> edwarnicke: will do
15:36:25 <edwarnicke> yamahata: Many thanks :)
15:37:04 <regXboi> yamahata: actually, if you want to write the PS that cleans up all of those anti-patterns, I'll be happy to review and merge :)
15:37:15 <yamahata> done
15:37:32 <yamahata> regXboi: got it.
15:38:26 <regXboi> #link https://git.opendaylight.org/gerrit/#/c/26711/  BGPVPN: Added yang, api and transcriber for BGPVPN.
15:38:43 <regXboi> I'm going to -1 that one because of the fact that it adds to the technical debt on tests
15:38:57 <regXboi> sorry edwarnicke, but I've put too much blood to let people add things that might break :)
15:39:57 <regXboi> especially after make that mistake in Li and seeing the yang models that resulted :(
15:40:42 <vthapar> regXboi: not looking for merge yet on https://git.opendaylight.org/gerrit/#/c/26711/  BGPVPN: Added yang, api and transcriber for BGPVPN. mainly interested in review comments for now.
15:40:58 <regXboi> ok... my big one is the IT cases
15:41:05 <vthapar> and any inputs on what more needs to be done to get them merge ready
15:41:06 <regXboi> but I'll look at the rest as well
15:41:21 <edwarnicke> regXboi: You are enough of a pragmatist that I tend to defer to you there... but reserve the right to make snarky comments ;)
15:41:28 <vthapar> np, wanted to wait for comments on basic things before adding ITs
15:41:30 <regXboi> that's expected
15:41:37 <regXboi> edwarnicke: that's expected
15:41:57 <edwarnicke> vthapar: https://git.opendaylight.org/gerrit/#/c/26856/ <- if this looks reasonable to you +1 so the releng/builder guys will merge
15:42:09 <regXboi> vthapar: my experience is that building the IT and getting it to run tends to uncover problems in the other code
15:42:42 <edwarnicke> vthapar: Courtesy check as I think this mostly impacts you, would you take a quick look at : https://git.opendaylight.org/gerrit/#/c/26581/
15:42:42 <vthapar> edwanicke: done.
15:42:50 <vthapar> edwarnicke: done
15:43:10 <edwarnicke> vthapar: So does https://git.opendaylight.org/gerrit/#/c/26581/ look good to you?
15:43:12 <vthapar> regXboi: ack.
15:43:49 <regXboi> vthapar: I say this because the transcriber class *looks* right on a read through
15:43:56 <vthapar> edwarnicke: 26581 is VPNAAS, bgpvpn is different. so can't comment on that.
15:43:56 <regXboi> but you won't really know until you try and use it
15:44:08 <edwarnicke> vthapar: Ah... apologies
15:44:18 <odp-gerritbot> A change was merged to neutron: Remove ConcurrentHashMaps from VPNaaS Interface classes  https://git.opendaylight.org/gerrit/26581
15:44:23 <odp-gerritbot> A change was merged to neutron: Remove use of ConcurrentHashMap from FWaaS transcriber  https://git.opendaylight.org/gerrit/26577
15:44:44 <vthapar> regXboi: wanted any comments that may result in change of yang or transcriber before adding IT.
15:44:49 <odp-gerritbot> A change was merged to neutron: Remove ConcurrentHashMaps from LBaaS classes  https://git.opendaylight.org/gerrit/26578
15:44:59 <regXboi> vthapar: unforunately, it's the other way around
15:45:06 <regXboi> IT tends to show you where your model is broken
15:45:07 <odp-gerritbot> A change was merged to neutron: Remove security group and security rule ConcurrentHashMaps  https://git.opendaylight.org/gerrit/26554
15:45:13 * yamahata wondering if we have time on odl driver rewrite...
15:45:14 <odp-gerritbot> A change was merged to neutron: Remove Metering ConcurrentHashMaps  https://git.opendaylight.org/gerrit/26551
15:45:22 <edwarnicke> regXboi: I think that's it :)
15:45:29 <vthapar> regXboi: aha, np. I'll add ITs in next patchset.
15:45:31 <regXboi> edwanrike: ack it is
15:45:36 <regXboi> ouch
15:45:39 <regXboi> edwarnicke: it is
15:45:51 <regXboi> and we are now in a position to seriously consider re-versioning yang!
15:46:07 <regXboi> but let's get yamahata's ip prefixes in first :)
15:46:21 <edwarnicke> :)
15:46:25 <regXboi> and any others we need
15:46:44 <regXboi> and I now get to figure out the md-sal dummy provider
15:47:10 <regXboi> #link https://git.opendaylight.org/gerrit/#/c/23594/ Proposed Neutron network to device map model
15:47:18 <regXboi> we need to spend some more quality time on this one
15:47:34 <regXboi> do we have enough time today or should we move it to the top of the list for next week?
15:47:43 <yamahata> I think it needs align with openstack neutron side.
15:48:14 <yamahata> I'd like to start to discuss the next topic this week.
15:48:20 <yamahata> even with 10min
15:48:22 <regXboi> yamahata: agreed
15:48:46 <regXboi> #action regXboi to schedule network to device map patch set at top of list for 9/18 meeting
15:48:52 <regXboi> and it will be so done
15:49:03 <regXboi> let's talk about the ODL driver
15:49:12 <regXboi> #topic ML2 ODL driver rewrite
15:49:24 <regXboi> I think the agenda is out of date
15:50:00 <yamahata> A WIP patch is published at  https://review.openstack.org/#/c/222409/
15:50:10 <yamahata> and it addresses request reordering and neutron HA.
15:50:36 <yamahata> The idea seems very promising because it directly addresses some of issues.
15:50:54 <regXboi> #link https://review.openstack.org/#/c/222409/ Opendaylight driver refactor to handle out of sync issues [WiP]
15:51:07 <yamahata> On the other hand, I suppose we should consider protocol change/enhancement for scalability and latency
15:51:13 <yamahata> They are ortogonal topic
15:51:16 <yamahata> orthogonal
15:51:27 <edwarnicke> yamahata: That patch looks like a good idea as far as it goes
15:51:34 <yamahata> I think people have their own thoughs/ideas.
15:51:39 <john_a_joyce> yamahata: we should consider parrallel tracks I think
15:51:50 <yamahata> john_a_joyce: asomya ping?
15:51:51 <edwarnicke> yamahata: But it *looks* like its just handling those issues in a single control node (or did I miss something)
15:52:18 <john_a_joyce> We put that patch together as somethign that could be done relatively quickly - without too much overhaul
15:52:33 <yamahata> edwarnicke: yes to some extent. It uses db row locking to cordinate neutron HA.
15:52:42 <yamahata> It needs to be revised, though.
15:52:51 <john_a_joyce> edwarnicke: no we beleive ti handles multiple nodes as well
15:52:52 <asomya> edwarnicke: Each neutron instance in a single thread handling syncing atomic objects to ODL, can be run with multiple neutron processes/threads
15:53:17 <yamahata> john_a_joyce: Can you elaborate more?
15:53:23 <edwarnicke> asomya: And what happens if we have multiple control nodes on completely separate boxes?
15:54:03 <asomya> with the im;licit row locks and validation code, they will woek on separate sync items
15:54:17 <edwarnicke> asomya: Ah... cool :)
15:54:27 <john_a_joyce> yamahata: each thread regardless of node will always go tot he DB to get the state before sending
15:54:48 <john_a_joyce> as asomya pointed it it locks the row while doing so
15:54:50 <asomya> the validation needs to be fleshed out a bit more, e.g. port creates will not be valid if there are pending network/subnet creates ot updates
15:54:54 <yamahata> john_a_joyce: There still remains race
15:55:04 <yamahata> Unfortunately we shouldn't depend on row locking
15:55:26 <yamahata> because Neutron rewrite dbcode to remove row locking to support Gallerra db backend
15:55:34 <edwarnicke> john_a_joyce: How are you tracking which changes have been sent to preserve global order of events across multiple nodes?
15:55:51 <yamahata> Maybe we can use locking at first. then rewrite it later.
15:56:13 <asomya> edwarnicke: the events have states tracked in the DB
15:56:16 <john_a_joyce> edwarnicke: we are not trying to preserve global order - we will retry
15:56:37 <asomya> with proper validation we wouldn't need to preserve order
15:56:43 <edwarnicke> john_a_joyce: OK... so if you are not preserving global order, it would be possible for a port create to come to ODL before we had received the network create?
15:57:13 <john_a_joyce> edwarnicke: that is true - but both would be on a retry queue
15:57:20 <edwarnicke> colindixon: You around?
15:57:32 * edwarnicke needs our resident distributed systems guru, colindixon
15:57:39 <asomya> edwarnicke: see reply above, the event validation code kicks in that prevents port creates/updates from being sent if there are pending operations on the asociated network or subnet
15:58:02 <edwarnicke> asomya: Cool... so you aren't preserving total order, but you are preserving order of dependencies
15:58:12 <edwarnicke> So if you have events A,B,C,D, and D depends on A
15:58:18 <john_a_joyce> edwarnicke: we are considering adding sequence or timestamps to help ensure the order - but a bit TBD still
15:58:30 <edwarnicke> You don't guarantee that order (A,B,C,D) ... but you *do* guarantee that D happens after A?
15:58:37 <asomya> so long as A is in processing or pending state, D will not be processed
15:58:44 <edwarnicke> asomya: Cool :)
15:58:57 <edwarnicke> At first blush that sounds like a smart comprimise to a hard problem :)
16:00:19 <edwarnicke> And you are using row locks then to make sure that two nodes don't send the same event?
16:00:29 <asomya> edwarnicke: correct
16:00:43 <john_a_joyce> maybe everyone can have a closer look and provide their comments on the change
16:01:47 <asomya> and we do track timestampe for when an event was created/last retried if we want to implement something like a strict ordered global sync later
16:02:13 <edwarnicke> asomya: Question... how do you handle the initial data dump?
16:02:44 <asomya> edwarnicke: i don't at the moment.. the initial code is only for atomic events
16:02:56 <yamahata> that's the issue. One simple idea is to use bulk create/update
16:03:00 <edwarnicke> asomya: Do you have thoughts there?
16:03:31 <asomya> edwarnicke: one thing we would need is some sort of feedback from ODL on what things are out of sync
16:03:49 <asomya> and only sync stuff that is out of sync instead of replaying the entire journal
16:03:49 * regXboi notes we are past our allotted time, but I'm willing to let the conversation continue in the logs :)
16:04:23 <yamahata> One idea is to introduce sequence number and to track where ODL is
16:04:27 <asomya> with timestamps if we can get the last synced event timestamp from ODl then we can play it from then on
16:04:34 <edwarnicke> asomya: I'm not sure ODL can know what is out of sync, but you can ask us what we think we know ;)
16:05:10 <asomya> edwarnicke: yeah that's what i meant , sequence number of timestamp:)
16:05:13 <asomya> *or
16:05:38 <yamahata> Maybe we can add seqnum/timestamp to each rest request
16:05:51 <john_a_joyce> edwarnicke:  I would be reluctant to put the onus on the neutron side to determine ODL is out of sync
16:05:57 <edwarnicke> asomya: I am leary of timestamps, because I have seen that fail to much (time sync across nodes is not as reliable as you'd think)
16:06:17 <yamahata> openstack neutron can track those seqnum/timestamp of each resource
16:06:19 <edwarnicke> asomya: But sequence numbers are good :)
16:06:25 <john_a_joyce> yes timestamps are a bit problematic - especially with multi-node cases
16:06:31 <asomya> edwarnicke: agreed, sequence number seem better
16:06:38 * edwarnicke has the scars on timestamps cross nodes ;)
16:06:51 <edwarnicke> If sequence numbers are good enough for DNS... they are good enough for me :)
16:06:59 <edwarnicke> (used to know a lot about DNS inner operations ;) )
16:07:51 <edwarnicke> asomya: How are you thinking of tracking the sequence numbers on the OpenStack side?
16:08:01 <asomya> Db rows?
16:08:13 <asomya> add a sequence number column to every row?
16:08:19 <edwarnicke> asomya: OK... that will serialize all operations though...
16:08:28 <john_a_joyce> so the ODL side is willing to take on the changes for handling a sequence number?
16:08:31 <yamahata> we can introduce new table like uuid, resource type, seqnum
16:08:38 <edwarnicke> asomya: But if you are doing it per row thats not so bad
16:08:55 <edwarnicke> (as per row operations are going to be serialized anyway)
16:09:59 <asomya> if we want to minimize the amount of data sent across, per row might be good
16:10:26 <edwarnicke> asomya: I can see that (just thinking through things outloud... please don't read any of my questions as *objections* ;) )
16:11:05 <asomya> edwarnicke: you're welcome to object :) we want feedback from the ODL folks
16:11:37 <regXboi> #info lots of discussion here - see the logs
16:12:00 * regXboi notes the last is because we still have an open meeting running :)
16:12:04 <asomya> Should we start an email thread to keep this discussion going?
16:12:18 <yamahata> asomya: +1
16:12:34 <john_a_joyce> asomya: +1
16:13:05 <edwarnicke> asomya: john_a_joyce Please :)  neutron-dev@lists.opendaylight.org :)
16:13:12 <asomya> ok i'll start a thread with the initlal plan
16:13:18 <edwarnicke> Many many thanks :)
16:14:00 <john_a_joyce> thanks everyone for the comments
16:15:27 <regXboi> are we done then?
16:15:33 <edwarnicke> john_a_joyce: Thank *you* guys for doing the work :)
16:15:52 <john_a_joyce> asomya did most of it
16:16:11 <john_a_joyce> I think we take this topic on to the thread
16:16:40 <asomya> sounds good
16:18:02 <regXboi> #endmeeting