16:03:32 <yamahata> #startmeeting neutron_northbound 16:03:32 <odl_meetbot> Meeting started Mon Dec 5 16:03:32 2016 UTC. The chair is yamahata. Information about MeetBot at http://ci.openstack.org/meetbot.html. 16:03:32 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:03:32 <odl_meetbot> The meeting name has been set to 'neutron_northbound' 16:03:41 <yamahata> #chair mkolesni 16:03:41 <odl_meetbot> Current chairs: mkolesni yamahata 16:03:49 <yamahata> #topic agenda bashing and roll call 16:03:55 <yamahata> #info yamahata 16:03:57 <mkolesni> #info mkolesni 16:04:23 <yamahata> #link https://wiki.opendaylight.org/view/NeutronNorthbound:Meetings 16:04:50 <yamahata> I updated the bugs and patches. 16:05:00 <mkolesni> ok 16:05:10 <yamahata> any additional topics to discuss today? 16:05:41 <mkolesni> no 16:05:49 <yamahata> I have one additional. stable branch maintenance. 16:06:18 <mkolesni> ok 16:06:42 <yamahata> #topic Announcements 16:06:57 <yamahata> Ocata-2 is Dec 12-16 16:07:05 <yamahata> #link https://releases.openstack.org/ocata/schedule.html 16:07:34 <yamahata> openstack ptg project team gathering is Feb 20-24 16:07:43 <yamahata> #link https://www.openstack.org/ptg/ 16:07:54 <yamahata> It has seat limit of 500 people. 16:08:22 <mkolesni> im probably not going to come 16:08:27 <yamahata> I heard last week it's about 300 left 16:08:54 <mkolesni> are you going? 16:09:15 <yamahata> Probably yes. I'm requesting for travel budget now. 16:09:42 <yamahata> It's in atlanta GA usa. 16:10:27 <yamahata> any other announcement? 16:11:03 <yamahata> Oh, opendaylight M2 offset 1 is Dec 1. 16:11:09 <yamahata> s/is/was/ 16:11:56 <yamahata> let's move on 16:12:06 <yamahata> #topic action items from last meeting 16:12:29 <yamahata> backport to stable/branch 16:12:49 <yamahata> Now I think stable/newton is in good shape. 16:13:02 <yamahata> also is stable/mitaka. 16:13:19 <mkolesni> yes seems to be in good shape 16:13:24 <yamahata> For now proposed patches were merged. 16:13:28 <mkolesni> was 3.1.1 released? 16:13:54 <yamahata> Not yet. You're fine. I'll request 3.1.1. release 16:14:03 <yamahata> #action yamahata release 3.1.1 16:14:45 <mkolesni> cool 16:14:49 <yamahata> next action item is review https://review.openstack.org/#/c/401124 16:15:02 <yamahata> The goal is to make functional test pass. 16:15:47 <yamahata> 401124 is work around and 404525 is fix 16:16:06 <yamahata> So I think we can merge 404525. mkolesni what do you think? 16:16:29 * mkolesni looking 16:17:29 <mkolesni> ok ill review it tomorrow 16:17:41 <mkolesni> #action mkolesni review 404525 16:17:53 <yamahata> Okay. Now Rui is also looking into make fullstack test run without devstack. 16:18:14 <yamahata> #link https://review.openstack.org/#/c/405052/ 16:18:14 <yamahata> WIP:fullstack - use functional scripts to install ODL 16:18:26 <mkolesni> ok great, maybe she can base on your pacth 16:18:28 <mkolesni> patch 16:18:48 <yamahata> the last action item, bug https://bugs.opendaylight.org/show_bug.cgi?id=7256 16:19:22 <yamahata> networking-odl side is fixed by 404003. 16:19:38 <mkolesni> yes i will make sure we get someone to fix it 16:19:46 <yamahata> For ODL side, Michael Vorburger is assigned to it. 16:20:09 <yamahata> Oops. He just gave the comment. 16:20:36 <mkolesni> #action mkolesni make sure 7256 on ODL Neutron Northbound is fixed 16:21:18 <yamahata> okay next topic 16:21:29 <yamahata> #topic carbon/ocata planning 16:22:04 <yamahata> The carbon cycle is short. so we need narrowed plan. 16:22:31 <yamahata> One forcus is to complete unfinished features. 16:23:02 <yamahata> Other focus is to enable those features by default and make them tested with openstack CI. 16:23:28 <yamahata> enable ODL new netvirt and make it tested with openstack ci. 16:23:44 <yamahata> make v2 driver/pseudo agent port binding enabled by default 16:24:13 <yamahata> I tested them a bit. 16:24:14 <mkolesni> i will be trying to fix up networking-odl to make v2 driver the default for ocata 16:24:27 <mkolesni> i found a bug while running tempest with v2 16:24:30 <mkolesni> #link https://review.openstack.org/406968 Add seqnum to dependency checks 16:24:38 <mkolesni> i sent a fix that should fix this 16:24:41 <yamahata> Cool. 16:24:59 <yamahata> #link https://review.openstack.org/#/c/382597/ DO NOT REVIEW/MERGE: test v2 driver 16:25:11 <mkolesni> generally i saw lots of rows stuck in "pending" 16:25:26 <mkolesni> because of circular dependency checks 16:25:32 <yamahata> It showed v2driver is very unstable. it results in failure with poenstack ci. 16:25:45 <mkolesni> yes im following that patch 16:25:53 <yamahata> Cool. 16:26:08 <mkolesni> also trying to recreate the same run locally to get the same results 16:26:44 <mkolesni> currently locally i see much more failures but probably im not configuring devstack/tempest like the gate 16:26:48 <mkolesni> so ill be working on that 16:27:41 <yamahata> Great. right now gate executes tests serially. Make it run in parallel would raise more issue. 16:28:16 <yamahata> I filed https://bugs.launchpad.net/networking-odl/+bug/1646681 16:28:33 <yamahata> I noticed it by looking at the log 16:28:34 <mkolesni> yes i think the parallel makes the circular dependencies issues pop up 16:28:49 <mkolesni> because of the volume of requests in a short time 16:28:58 <yamahata> #link https://bugs.launchpad.net/networking-odl/+bug/1646681 v2 driver: too many skipping row on same resource in short period Edit 16:29:30 <mkolesni> yes 16:29:45 <mkolesni> probably this is the same issue i witnessed 16:29:46 <yamahata> I'm guessing it happens only when there is only one op left in journal. So it's not cirtical. 16:30:21 <yamahata> I haven't dug into it so deeply. 16:30:49 <mkolesni> actually i saw more than 100 rows there 16:31:01 <mkolesni> and the driver just running over them endlessly 16:32:03 <yamahata> that sounds plausible. 16:32:03 <john_a_joyce> Mike - Isaku are aware of anyone using the V2 driver in production that is seeing these issues? 16:32:35 <mkolesni> hi John! no im not aware of any use of this in production yet 16:32:45 <yamahata> No at the moment. 16:32:53 <john_a_joyce> We are using the V2 design on a Cisco specific controller and aren't seeing these issues 16:32:56 <mkolesni> or not production for that instance 16:32:58 <yamahata> ODL new netvirt people are testing it with ODL CI. 16:33:09 <john_a_joyce> it is in a production environment 16:33:24 <yamahata> They say v2 driver is stable with ODL CI. 16:33:30 <mkolesni> i think it happens where theres a lot of requests simultaneously 16:33:40 <yamahata> But I guess it's not production environment. 16:33:44 <john_a_joyce> ok - I will look closely at these bugs and make sure we didn't add anything in our environment 16:33:47 <mkolesni> not sure if this happens frequently in production 16:34:04 <john_a_joyce> generally our driver is behind the v2 driver by a few patches 16:34:37 <john_a_joyce> glad to see the testing is expanding though 16:34:49 <yamahata> In high load case, neutron api process speed can be > journal process speed. 16:35:05 <yamahata> In this case, neutron api process needs to be throttled somehow. 16:35:44 <john_a_joyce> that might be the difference our production environment doesn't pound the neutron API 16:35:52 <mkolesni> probably 16:36:06 <mkolesni> im not sure throttling is a good solution it wont scale 16:36:07 <yamahata> We don't have it right now. journal prcess speed is determined by ODL. 16:36:10 <john_a_joyce> so the API load is moderate at best 16:36:15 <mkolesni> better fix the bugs :) 16:36:49 <yamahata> mkolesni: Yeah bug should be fixed first. Please go ahead :) 16:36:51 <john_a_joyce> yeah throttling usually just moves the bug somewhere else 16:37:29 <mkolesni> as i said sent a patch for the circular dependency stuff we will see if it improves the behavior 16:38:05 <yamahata> #topic patches/bugs 16:38:44 <yamahata> any other bugs/patches? 16:39:05 <mkolesni> none from me 16:39:34 <yamahata> #link http://status.openstack.org/openstack-health/#/g/project/openstack~2Fnetworking-odl 16:40:04 <yamahata> we have priodic job in openstack CI. I noticed periodic job with liberty fails to run 16:40:25 <yamahata> peridodic job with newton is able to run it, but results in test case failure. 16:40:52 <yamahata> Since liberty is about to be end-of-life, it's okay to stop liberty periodic job. 16:41:01 <yamahata> What should we do with mitaka branch? and newton? 16:41:40 <yamahata> Just can we stop those periodic job? or do we want to fix mitaka branch? do we want to add newton periodic job? 16:41:53 <yamahata> #link http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-networking-odl-beryllium-snapshot-liberty 16:42:02 <yamahata> #link http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-networking-odl-boron-snapshot-liberty 16:42:09 <yamahata> #link http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-networking-odl-boron-snapshot-liberty 16:42:32 <mkolesni> whats the point of the periodic job? 16:43:00 <mkolesni> to test against ODL stablke changes once in a while? 16:43:03 <mkolesni> stable 16:43:10 <yamahata> the goal is to test with ODL master head in order to detect breakage. 16:43:21 <yamahata> ODL master + networking-odl stable 16:43:43 <mkolesni> seems reasonable then to have these jobs for the stable branches we support 16:43:46 <yamahata> oh no. ODL stable + networking-odl stable 16:44:13 <mkolesni> unless we have such checks on ODL side already 16:44:45 <yamahata> opendaylight CI has similar. and also OPNFV is testing with stable branch. 16:45:21 <mkolesni> i guess they make more sense to be on ODL side 16:45:28 <yamahata> I'm not monitoring their results, though. 16:45:31 <mkolesni> there is more potential for breakage there 16:46:29 <yamahata> I see. Given the current situation, it would make sense to delete those periodic job. 16:46:47 <yamahata> Anyway if we think it's necessary, we can re-add them again. 16:47:30 <mkolesni> ok 16:47:40 <yamahata> #action yamahata delete odl periodic job from openstack CI. 16:47:54 <yamahata> any other patches/bugs? 16:49:53 <yamahata> BTW Right now my team is trying to fix multinode CI. 16:50:21 <yamahata> I'm wondering to add vagrant file to setup multinode case to the repo because it's difficult to setup its environment locally. 16:50:30 <yamahata> vagrant files for it would help. 16:50:46 <yamahata> anything else or we can move to open mike 16:51:04 <mkolesni> nothing from me 16:51:11 <yamahata> #topic open mike 16:51:24 <yamahata> let me discuss on stable branch maintenance. 16:51:57 <yamahata> In recent discussion on netvirt mailing list, there seems requirement for stable/newton branch to have backports. 16:52:25 <yamahata> mkolesni: do you have any interest? 16:52:25 <mkolesni> dont think im on that list 16:52:30 <mkolesni> what do they want backported? 16:52:58 <yamahata> #link https://lists.opendaylight.org/mailman/listinfo/netvirt-dev 16:53:34 <yamahata> At the moment, they don't have concrete ones. but furture maintenance. 16:53:51 <yamahata> e.g. bug fixes for v2 driver, pseudo agent port binding 16:54:15 <yamahata> As we'll migrate to new netvirt, netwokring-odl may have more bug fixes realted to it. 16:54:26 <mkolesni> well lets see what pops up and tackle it as it arrives 16:54:38 <mkolesni> i thought there something concrete being discussed 16:55:04 <yamahata> ok. 16:55:47 <mkolesni> you have something in mind? 16:55:47 <yamahata> any other topic to discuss? 16:56:00 <yamahata> For now no. 16:56:06 <mkolesni> ok 16:56:29 <john_a_joyce> nothing from me 16:56:46 <yamahata> next week Dec 12, we will have meeting. 16:57:04 <yamahata> On Dec 19 I can host it. 16:57:14 <yamahata> We will skip Dec 26 and Jan 2. 16:57:20 <mkolesni> ok 16:57:49 <yamahata> I'll give you 3min back 16:57:53 <yamahata> thank you everyone 16:57:55 <mkolesni> ok :) 16:57:59 <mkolesni> thanks Isaku 16:58:07 <yamahata> #topic cookies 16:58:11 <yamahata> #endmeeting