15:01:19 <regXboi> #startmeeting neutron_northbound 15:01:19 <odl_meetbot> Meeting started Fri Sep 11 15:01:19 2015 UTC. The chair is regXboi. Information about MeetBot at http://ci.openstack.org/meetbot.html. 15:01:19 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:19 <odl_meetbot> The meeting name has been set to 'neutron_northbound' 15:01:27 <regXboi> #topic agenda bash and roll call 15:01:31 <regXboi> #info regXboi 15:01:42 <yamahata> #info yamahata 15:01:56 <regXboi> #info apologies for no meeting last week, there were a dirth of committers 15:02:26 <regXboi> #link https://wiki.opendaylight.org/view/NeutronNorthbound:Meetings agenda from last week, which we'll start with 15:02:43 <edwarnicke> #info edwarnicke 15:02:49 <regXboi> and we have a quorum 15:03:16 <regXboi> #topic action items 15:03:31 <regXboi> item: edwarnicke, flaviof, regXboi to look at bug 3990 15:03:47 <regXboi> I haven't looked at this, so I think we'll have to carry it as still open, what say you edwarnicke? 15:03:51 <regXboi> #chair edwarnicke 15:03:51 <odl_meetbot> Current chairs: edwarnicke regXboi 15:04:38 <edwarnicke> regXboi: I haven't looked to closely at this either... but what's curious about it is that we use openjdk in the Jenkins Jobs 15:04:54 <regXboi> yeah, that's why I was a bit surprised 15:05:33 <regXboi> ok, let's put it back on the AI list 15:05:36 <regXboi> #action edwarnicke, flaviof, regXboi to look at bug 3990 15:05:46 <regXboi> item: yamahata to check and comment on bug 4026 15:06:17 <edwarnicke> regXboi: If its a race condition we could just be getting lucky in the Jenkins Builds 15:06:20 <yamahata> I haven't. will look into it. 15:06:30 <regXboi> ok, back to AI for that 15:06:30 <yamahata> re 4026 15:06:37 <regXboi> edwarnicke: ack, yamahata: thx 15:06:45 <regXboi> #action yamahata to check and comment on bug 4026 15:06:52 <regXboi> item regXboi to add IT test case for bug 4027 15:06:58 <regXboi> that patch has merged, so that item is done :) 15:07:07 <regXboi> item: yamahata to look at bug 4043 15:07:33 <yamahata> re 4043, the cause is identified, and the patches were uploaded. 15:07:40 <edwarnicke> yamahata: link? 15:07:53 * yamahata looking for it 15:07:55 <regXboi> #link https://git.opendaylight.org/gerrit/#/c/26266/ PS for 4043 15:08:12 <yamahata> thanks 15:08:16 <regXboi> although honestly, we need to fixup ALL of the places where we store CIDR blocks 15:08:43 <yamahata> and also 26267, I found the similar issue. 15:08:51 <regXboi> so... I'm really uncomfortable merging those patch sets 15:09:12 <yamahata> regXboi: Are you against the patch? or merge conflict issue? 15:09:16 <edwarnicke> https://git.opendaylight.org/gerrit/#/c/26266/ looks good to me, any idea why flaviof didn't merge? 15:09:18 <regXboi> neither of them have test cases so that jenkins can actually verify the change 15:09:27 <regXboi> edwarnicke: ^^^^^^ 15:09:46 <regXboi> i.e. we *think* it is correct, but how do we *know* it is correct 15:09:49 <yamahata> Ah, sure. I'll add test cases. 15:09:54 * edwarnicke is less of a unit test zealot... but then regXboi is the one who bled to get our testing to where its at 15:10:08 <regXboi> regXboi has watched too many regressions happen 15:10:14 <regXboi> and I actually don't think this is UT 15:10:17 <regXboi> I think it is IT 15:10:18 <regXboi> :) 15:10:20 <edwarnicke> There are those who maintain I am a regression ;) 15:10:28 * regXboi isn't going to touch that one 15:10:54 <regXboi> yamahata: please add the test cases to each and ping me when you have them done and I'll review and merge them in 15:11:00 <regXboi> in the meantime 15:11:06 <yamahata> regXboi: will do 15:11:16 <regXboi> #action yamahata to add test cases to PS 26266 and 26267 15:11:29 <regXboi> #action regXboi to review and merge 26266 and 26267 once test cases are in place 15:12:07 <regXboi> speaking of which - we have a bunch of things to merge to prep tht 15:13:43 <regXboi> next item: yamahata to fix up https://git.opendaylight.org/gerrit/#/c/25500/ 15:14:03 <regXboi> yamahata: actually, I have a lot of PS in flight that have that 15:14:14 <yamahata> I didn't have time to fix 25500. 15:14:27 <regXboi> yamahata: no worries, we'll carry it as an AI 15:14:46 <regXboi> #action yamahata to fix up https://git.opendaylight.org/gerrit/#/c/25500/ 15:14:50 <regXboi> simply because 15:14:58 <regXboi> item regXboi to look at adding out of order IT tests to catch other issues like bug 4188 15:15:03 <regXboi> I haven't had time to do that either 15:15:04 <regXboi> so 15:15:07 <regXboi> #action regXboi to look at adding out of order IT tests to catch other issues like bug 4188 15:15:17 <regXboi> and last: everybody to think about how to deal with issues like https://bugs.opendaylight.org/show_bug.cgi?id=4188 15:15:27 <regXboi> I think that one needs to be part of the later discussion 15:15:57 <regXboi> under the heading of open patch sets 15:16:13 <regXboi> there are a bunch related to pulling out the CHMs 15:16:40 <regXboi> yamahata-san: I saw your -1 and answered - I really would rather clean up all the patterns in a single PS after everything merges 15:17:03 <yamahata> regXboi: I see, that's fine with me. 15:17:11 <regXboi> anything else or next topic? 15:17:23 <regXboi> #topic Berylliym 15:17:31 <regXboi> just an update here: 15:17:42 <regXboi> we have dependencies from the following downstream groups (so far): VTN, OVSDB, BGPVPN, LispFlowMapping, GBP 15:18:10 <regXboi> #info we have dependencies from the following downstream groups (so far): VTN, OVSDB, BGPVPN, LispFlowMapping, GBP 15:18:28 <regXboi> #info BGPVPN and GBP need to configure their jjb correctly so that the dependency appears in our merge 15:18:47 <regXboi> edwarnicke: can I task you with helping the bgpvpn and gbp folks get that done? 15:19:39 <regXboi> edwarnicke: ping? 15:21:53 <regXboi> ok, I'll take silence as a yes :) 15:22:28 <regXboi> #action edwarnicke to work with BGPVPN and GBP projects about getting their dependency on NN into their JJB configs so that integration on merge is handled correctly 15:22:47 <regXboi> #topic Other patch sets 15:23:10 <regXboi> I'm not going to put the ConcurrentHashMaps in here because that is (in my mind at least) just cleaning up 15:23:58 <odp-gerritbot> A change was merged to neutron: Migration to use MD-SAL Project https://git.opendaylight.org/gerrit/26353 15:25:36 <regXboi> #link https://git.opendaylight.org/gerrit/#/c/26128/ northbound: return 404 when deleting unexisting uuid 15:26:04 <regXboi> if we can get the merge conflict fixed, I can see that merging 15:26:25 <yamahata> I'll rebase them. 15:26:39 <regXboi> yamahata: you might want to wait until the CHMs all land 15:26:47 <regXboi> some of them do a bit of changing to the NN 15:27:10 <yamahata> regXboi: okay. I'll wait. Please let me know when done. 15:27:18 <regXboi> yamahata: ack 15:27:46 <regXboi> #link https://git.opendaylight.org/gerrit/#/c/26127/ northbound: introduce a base class 15:27:58 <edwarnicke> regXboi: Apologies... was briefly distracted, the bgpvpn and gbp guys need their jjbs to depend on neutron? Correct? 15:28:06 <regXboi> edwarnicke: yes 15:28:09 <regXboi> you have an AI on that 15:28:25 <regXboi> and it look slike 26127 needs a rebase as well 15:29:29 <regXboi> #link https://git.opendaylight.org/gerrit/#/c/25332/ BUG #4143 : Transation failed with OptimisticLockFailedException 15:29:43 <regXboi> edwarnicke: any reason we can't merge ^^^^^ ? 15:29:54 <regXboi> oh 15:29:55 <regXboi> I see 15:29:59 <regXboi> transaction chains 15:30:07 <yamahata> If transaction chains is necessary, I'll work on it. 15:30:14 <yamahata> Should I convert it into chain? 15:30:21 <regXboi> yamahata: please and thank you 15:30:26 <yamahata> got it. 15:30:32 <edwarnicke> yamahata: I can advise on transaction chains :) 15:30:35 <regXboi> oh heck 15:30:39 <edwarnicke> yamahata: I've done them before :) 15:30:43 <edwarnicke> yamahata: Just ping me on IRC :) 15:30:47 <yamahata> edwarnicke: thanks. 15:30:47 <regXboi> edwarnicke: let's get the CHM patches all merged 15:31:15 <regXboi> there is a whole series of them 15:31:18 <regXboi> https://git.opendaylight.org/gerrit/#/c/26551/ 15:31:26 <regXboi> https://git.opendaylight.org/gerrit/#/c/26554/ 15:31:36 <regXboi> https://git.opendaylight.org/gerrit/#/c/26577/ 15:31:43 <regXboi> https://git.opendaylight.org/gerrit/#/c/26581/ 15:31:53 <regXboi> https://git.opendaylight.org/gerrit/#/c/26578/ 15:32:05 <yamahata> I'm catching up to review them. Not finished yet. 15:32:27 <regXboi> unfortunately, they get bigger as you go because they needed more changes to make work 15:32:28 <regXboi> :( 15:32:40 <regXboi> hey colindixon - you around? 15:32:42 <edwarnicke> regXboi: GBP jjb neutron dependency: https://git.opendaylight.org/gerrit/#/c/26855/ 15:32:53 <colindixon> regXboi: mostly 15:33:00 <edwarnicke> regXboi: vpnservice jjb neutron dependency: https://git.opendaylight.org/gerrit/#/c/26856/ 15:33:10 <regXboi> if so... those patches are why stable/li backport of yang is messy 15:33:38 <regXboi> edwarnicke: ack and I've +1'd both of them 15:33:56 <edwarnicke> yamahata: Are you cool with regXboi's response here: https://git.opendaylight.org/gerrit/#/c/26551/ 15:34:06 <edwarnicke> yamahata: and here: https://git.opendaylight.org/gerrit/#/c/26554/ 15:35:37 <regXboi> edwarnicke: scroll back to around 10:16 of this meeting for the answer 15:35:48 <regXboi> er 10:16 Central Time :) 15:35:59 <edwarnicke> yamahata: Could you un -1 then those two patches? 15:36:11 <yamahata> edwarnicke: will do 15:36:25 <edwarnicke> yamahata: Many thanks :) 15:37:04 <regXboi> yamahata: actually, if you want to write the PS that cleans up all of those anti-patterns, I'll be happy to review and merge :) 15:37:15 <yamahata> done 15:37:32 <yamahata> regXboi: got it. 15:38:26 <regXboi> #link https://git.opendaylight.org/gerrit/#/c/26711/ BGPVPN: Added yang, api and transcriber for BGPVPN. 15:38:43 <regXboi> I'm going to -1 that one because of the fact that it adds to the technical debt on tests 15:38:57 <regXboi> sorry edwarnicke, but I've put too much blood to let people add things that might break :) 15:39:57 <regXboi> especially after make that mistake in Li and seeing the yang models that resulted :( 15:40:42 <vthapar> regXboi: not looking for merge yet on https://git.opendaylight.org/gerrit/#/c/26711/ BGPVPN: Added yang, api and transcriber for BGPVPN. mainly interested in review comments for now. 15:40:58 <regXboi> ok... my big one is the IT cases 15:41:05 <vthapar> and any inputs on what more needs to be done to get them merge ready 15:41:06 <regXboi> but I'll look at the rest as well 15:41:21 <edwarnicke> regXboi: You are enough of a pragmatist that I tend to defer to you there... but reserve the right to make snarky comments ;) 15:41:28 <vthapar> np, wanted to wait for comments on basic things before adding ITs 15:41:30 <regXboi> that's expected 15:41:37 <regXboi> edwarnicke: that's expected 15:41:57 <edwarnicke> vthapar: https://git.opendaylight.org/gerrit/#/c/26856/ <- if this looks reasonable to you +1 so the releng/builder guys will merge 15:42:09 <regXboi> vthapar: my experience is that building the IT and getting it to run tends to uncover problems in the other code 15:42:42 <edwarnicke> vthapar: Courtesy check as I think this mostly impacts you, would you take a quick look at : https://git.opendaylight.org/gerrit/#/c/26581/ 15:42:42 <vthapar> edwanicke: done. 15:42:50 <vthapar> edwarnicke: done 15:43:10 <edwarnicke> vthapar: So does https://git.opendaylight.org/gerrit/#/c/26581/ look good to you? 15:43:12 <vthapar> regXboi: ack. 15:43:49 <regXboi> vthapar: I say this because the transcriber class *looks* right on a read through 15:43:56 <vthapar> edwarnicke: 26581 is VPNAAS, bgpvpn is different. so can't comment on that. 15:43:56 <regXboi> but you won't really know until you try and use it 15:44:08 <edwarnicke> vthapar: Ah... apologies 15:44:18 <odp-gerritbot> A change was merged to neutron: Remove ConcurrentHashMaps from VPNaaS Interface classes https://git.opendaylight.org/gerrit/26581 15:44:23 <odp-gerritbot> A change was merged to neutron: Remove use of ConcurrentHashMap from FWaaS transcriber https://git.opendaylight.org/gerrit/26577 15:44:44 <vthapar> regXboi: wanted any comments that may result in change of yang or transcriber before adding IT. 15:44:49 <odp-gerritbot> A change was merged to neutron: Remove ConcurrentHashMaps from LBaaS classes https://git.opendaylight.org/gerrit/26578 15:44:59 <regXboi> vthapar: unforunately, it's the other way around 15:45:06 <regXboi> IT tends to show you where your model is broken 15:45:07 <odp-gerritbot> A change was merged to neutron: Remove security group and security rule ConcurrentHashMaps https://git.opendaylight.org/gerrit/26554 15:45:13 * yamahata wondering if we have time on odl driver rewrite... 15:45:14 <odp-gerritbot> A change was merged to neutron: Remove Metering ConcurrentHashMaps https://git.opendaylight.org/gerrit/26551 15:45:22 <edwarnicke> regXboi: I think that's it :) 15:45:29 <vthapar> regXboi: aha, np. I'll add ITs in next patchset. 15:45:31 <regXboi> edwanrike: ack it is 15:45:36 <regXboi> ouch 15:45:39 <regXboi> edwarnicke: it is 15:45:51 <regXboi> and we are now in a position to seriously consider re-versioning yang! 15:46:07 <regXboi> but let's get yamahata's ip prefixes in first :) 15:46:21 <edwarnicke> :) 15:46:25 <regXboi> and any others we need 15:46:44 <regXboi> and I now get to figure out the md-sal dummy provider 15:47:10 <regXboi> #link https://git.opendaylight.org/gerrit/#/c/23594/ Proposed Neutron network to device map model 15:47:18 <regXboi> we need to spend some more quality time on this one 15:47:34 <regXboi> do we have enough time today or should we move it to the top of the list for next week? 15:47:43 <yamahata> I think it needs align with openstack neutron side. 15:48:14 <yamahata> I'd like to start to discuss the next topic this week. 15:48:20 <yamahata> even with 10min 15:48:22 <regXboi> yamahata: agreed 15:48:46 <regXboi> #action regXboi to schedule network to device map patch set at top of list for 9/18 meeting 15:48:52 <regXboi> and it will be so done 15:49:03 <regXboi> let's talk about the ODL driver 15:49:12 <regXboi> #topic ML2 ODL driver rewrite 15:49:24 <regXboi> I think the agenda is out of date 15:50:00 <yamahata> A WIP patch is published at https://review.openstack.org/#/c/222409/ 15:50:10 <yamahata> and it addresses request reordering and neutron HA. 15:50:36 <yamahata> The idea seems very promising because it directly addresses some of issues. 15:50:54 <regXboi> #link https://review.openstack.org/#/c/222409/ Opendaylight driver refactor to handle out of sync issues [WiP] 15:51:07 <yamahata> On the other hand, I suppose we should consider protocol change/enhancement for scalability and latency 15:51:13 <yamahata> They are ortogonal topic 15:51:16 <yamahata> orthogonal 15:51:27 <edwarnicke> yamahata: That patch looks like a good idea as far as it goes 15:51:34 <yamahata> I think people have their own thoughs/ideas. 15:51:39 <john_a_joyce> yamahata: we should consider parrallel tracks I think 15:51:50 <yamahata> john_a_joyce: asomya ping? 15:51:51 <edwarnicke> yamahata: But it *looks* like its just handling those issues in a single control node (or did I miss something) 15:52:18 <john_a_joyce> We put that patch together as somethign that could be done relatively quickly - without too much overhaul 15:52:33 <yamahata> edwarnicke: yes to some extent. It uses db row locking to cordinate neutron HA. 15:52:42 <yamahata> It needs to be revised, though. 15:52:51 <john_a_joyce> edwarnicke: no we beleive ti handles multiple nodes as well 15:52:52 <asomya> edwarnicke: Each neutron instance in a single thread handling syncing atomic objects to ODL, can be run with multiple neutron processes/threads 15:53:17 <yamahata> john_a_joyce: Can you elaborate more? 15:53:23 <edwarnicke> asomya: And what happens if we have multiple control nodes on completely separate boxes? 15:54:03 <asomya> with the im;licit row locks and validation code, they will woek on separate sync items 15:54:17 <edwarnicke> asomya: Ah... cool :) 15:54:27 <john_a_joyce> yamahata: each thread regardless of node will always go tot he DB to get the state before sending 15:54:48 <john_a_joyce> as asomya pointed it it locks the row while doing so 15:54:50 <asomya> the validation needs to be fleshed out a bit more, e.g. port creates will not be valid if there are pending network/subnet creates ot updates 15:54:54 <yamahata> john_a_joyce: There still remains race 15:55:04 <yamahata> Unfortunately we shouldn't depend on row locking 15:55:26 <yamahata> because Neutron rewrite dbcode to remove row locking to support Gallerra db backend 15:55:34 <edwarnicke> john_a_joyce: How are you tracking which changes have been sent to preserve global order of events across multiple nodes? 15:55:51 <yamahata> Maybe we can use locking at first. then rewrite it later. 15:56:13 <asomya> edwarnicke: the events have states tracked in the DB 15:56:16 <john_a_joyce> edwarnicke: we are not trying to preserve global order - we will retry 15:56:37 <asomya> with proper validation we wouldn't need to preserve order 15:56:43 <edwarnicke> john_a_joyce: OK... so if you are not preserving global order, it would be possible for a port create to come to ODL before we had received the network create? 15:57:13 <john_a_joyce> edwarnicke: that is true - but both would be on a retry queue 15:57:20 <edwarnicke> colindixon: You around? 15:57:32 * edwarnicke needs our resident distributed systems guru, colindixon 15:57:39 <asomya> edwarnicke: see reply above, the event validation code kicks in that prevents port creates/updates from being sent if there are pending operations on the asociated network or subnet 15:58:02 <edwarnicke> asomya: Cool... so you aren't preserving total order, but you are preserving order of dependencies 15:58:12 <edwarnicke> So if you have events A,B,C,D, and D depends on A 15:58:18 <john_a_joyce> edwarnicke: we are considering adding sequence or timestamps to help ensure the order - but a bit TBD still 15:58:30 <edwarnicke> You don't guarantee that order (A,B,C,D) ... but you *do* guarantee that D happens after A? 15:58:37 <asomya> so long as A is in processing or pending state, D will not be processed 15:58:44 <edwarnicke> asomya: Cool :) 15:58:57 <edwarnicke> At first blush that sounds like a smart comprimise to a hard problem :) 16:00:19 <edwarnicke> And you are using row locks then to make sure that two nodes don't send the same event? 16:00:29 <asomya> edwarnicke: correct 16:00:43 <john_a_joyce> maybe everyone can have a closer look and provide their comments on the change 16:01:47 <asomya> and we do track timestampe for when an event was created/last retried if we want to implement something like a strict ordered global sync later 16:02:13 <edwarnicke> asomya: Question... how do you handle the initial data dump? 16:02:44 <asomya> edwarnicke: i don't at the moment.. the initial code is only for atomic events 16:02:56 <yamahata> that's the issue. One simple idea is to use bulk create/update 16:03:00 <edwarnicke> asomya: Do you have thoughts there? 16:03:31 <asomya> edwarnicke: one thing we would need is some sort of feedback from ODL on what things are out of sync 16:03:49 <asomya> and only sync stuff that is out of sync instead of replaying the entire journal 16:03:49 * regXboi notes we are past our allotted time, but I'm willing to let the conversation continue in the logs :) 16:04:23 <yamahata> One idea is to introduce sequence number and to track where ODL is 16:04:27 <asomya> with timestamps if we can get the last synced event timestamp from ODl then we can play it from then on 16:04:34 <edwarnicke> asomya: I'm not sure ODL can know what is out of sync, but you can ask us what we think we know ;) 16:05:10 <asomya> edwarnicke: yeah that's what i meant , sequence number of timestamp:) 16:05:13 <asomya> *or 16:05:38 <yamahata> Maybe we can add seqnum/timestamp to each rest request 16:05:51 <john_a_joyce> edwarnicke: I would be reluctant to put the onus on the neutron side to determine ODL is out of sync 16:05:57 <edwarnicke> asomya: I am leary of timestamps, because I have seen that fail to much (time sync across nodes is not as reliable as you'd think) 16:06:17 <yamahata> openstack neutron can track those seqnum/timestamp of each resource 16:06:19 <edwarnicke> asomya: But sequence numbers are good :) 16:06:25 <john_a_joyce> yes timestamps are a bit problematic - especially with multi-node cases 16:06:31 <asomya> edwarnicke: agreed, sequence number seem better 16:06:38 * edwarnicke has the scars on timestamps cross nodes ;) 16:06:51 <edwarnicke> If sequence numbers are good enough for DNS... they are good enough for me :) 16:06:59 <edwarnicke> (used to know a lot about DNS inner operations ;) ) 16:07:51 <edwarnicke> asomya: How are you thinking of tracking the sequence numbers on the OpenStack side? 16:08:01 <asomya> Db rows? 16:08:13 <asomya> add a sequence number column to every row? 16:08:19 <edwarnicke> asomya: OK... that will serialize all operations though... 16:08:28 <john_a_joyce> so the ODL side is willing to take on the changes for handling a sequence number? 16:08:31 <yamahata> we can introduce new table like uuid, resource type, seqnum 16:08:38 <edwarnicke> asomya: But if you are doing it per row thats not so bad 16:08:55 <edwarnicke> (as per row operations are going to be serialized anyway) 16:09:59 <asomya> if we want to minimize the amount of data sent across, per row might be good 16:10:26 <edwarnicke> asomya: I can see that (just thinking through things outloud... please don't read any of my questions as *objections* ;) ) 16:11:05 <asomya> edwarnicke: you're welcome to object :) we want feedback from the ODL folks 16:11:37 <regXboi> #info lots of discussion here - see the logs 16:12:00 * regXboi notes the last is because we still have an open meeting running :) 16:12:04 <asomya> Should we start an email thread to keep this discussion going? 16:12:18 <yamahata> asomya: +1 16:12:34 <john_a_joyce> asomya: +1 16:13:05 <edwarnicke> asomya: john_a_joyce Please :) neutron-dev@lists.opendaylight.org :) 16:13:12 <asomya> ok i'll start a thread with the initlal plan 16:13:18 <edwarnicke> Many many thanks :) 16:14:00 <john_a_joyce> thanks everyone for the comments 16:15:27 <regXboi> are we done then? 16:15:33 <edwarnicke> john_a_joyce: Thank *you* guys for doing the work :) 16:15:52 <john_a_joyce> asomya did most of it 16:16:11 <john_a_joyce> I think we take this topic on to the thread 16:16:40 <asomya> sounds good 16:18:02 <regXboi> #endmeeting