16:03:32 <yamahata> #startmeeting neutron_northbound
16:03:32 <odl_meetbot> Meeting started Mon Dec  5 16:03:32 2016 UTC.  The chair is yamahata. Information about MeetBot at http://ci.openstack.org/meetbot.html.
16:03:32 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:03:32 <odl_meetbot> The meeting name has been set to 'neutron_northbound'
16:03:41 <yamahata> #chair mkolesni
16:03:41 <odl_meetbot> Current chairs: mkolesni yamahata
16:03:49 <yamahata> #topic agenda bashing and roll call
16:03:55 <yamahata> #info yamahata
16:03:57 <mkolesni> #info mkolesni
16:04:23 <yamahata> #link https://wiki.opendaylight.org/view/NeutronNorthbound:Meetings
16:04:50 <yamahata> I updated the bugs and patches.
16:05:00 <mkolesni> ok
16:05:10 <yamahata> any additional topics to discuss today?
16:05:41 <mkolesni> no
16:05:49 <yamahata> I have one additional. stable branch maintenance.
16:06:18 <mkolesni> ok
16:06:42 <yamahata> #topic Announcements
16:06:57 <yamahata> Ocata-2 is Dec 12-16
16:07:05 <yamahata> #link https://releases.openstack.org/ocata/schedule.html
16:07:34 <yamahata> openstack ptg project team gathering is Feb 20-24
16:07:43 <yamahata> #link https://www.openstack.org/ptg/
16:07:54 <yamahata> It has seat limit of 500 people.
16:08:22 <mkolesni> im probably not going to come
16:08:27 <yamahata> I heard last week it's about 300 left
16:08:54 <mkolesni> are you going?
16:09:15 <yamahata> Probably yes. I'm requesting for travel budget now.
16:09:42 <yamahata> It's in atlanta GA usa.
16:10:27 <yamahata> any other announcement?
16:11:03 <yamahata> Oh, opendaylight M2 offset 1 is Dec 1.
16:11:09 <yamahata> s/is/was/
16:11:56 <yamahata> let's move on
16:12:06 <yamahata> #topic action items from last meeting
16:12:29 <yamahata> backport to stable/branch
16:12:49 <yamahata> Now I think stable/newton is in good shape.
16:13:02 <yamahata> also is stable/mitaka.
16:13:19 <mkolesni> yes seems to be in good shape
16:13:24 <yamahata> For now proposed patches were merged.
16:13:28 <mkolesni> was 3.1.1 released?
16:13:54 <yamahata> Not yet. You're fine. I'll request 3.1.1. release
16:14:03 <yamahata> #action yamahata release 3.1.1
16:14:45 <mkolesni> cool
16:14:49 <yamahata> next action item is review https://review.openstack.org/#/c/401124
16:15:02 <yamahata> The goal is to make functional test pass.
16:15:47 <yamahata> 401124 is work around and 404525 is fix
16:16:06 <yamahata> So I think we can merge 404525. mkolesni what do you think?
16:16:29 * mkolesni looking
16:17:29 <mkolesni> ok ill review it tomorrow
16:17:41 <mkolesni> #action mkolesni review 404525
16:17:53 <yamahata> Okay. Now Rui is also looking into make fullstack test run without devstack.
16:18:14 <yamahata> #link https://review.openstack.org/#/c/405052/
16:18:14 <yamahata> WIP:fullstack - use functional scripts to install ODL
16:18:26 <mkolesni> ok great, maybe she can base on your pacth
16:18:28 <mkolesni> patch
16:18:48 <yamahata> the last action item, bug https://bugs.opendaylight.org/show_bug.cgi?id=7256
16:19:22 <yamahata> networking-odl side is fixed by 404003.
16:19:38 <mkolesni> yes i will make sure we get someone to fix it
16:19:46 <yamahata> For ODL side, Michael Vorburger  is assigned to it.
16:20:09 <yamahata> Oops. He just gave the comment.
16:20:36 <mkolesni> #action mkolesni make sure 7256 on ODL Neutron Northbound is fixed
16:21:18 <yamahata> okay next topic
16:21:29 <yamahata> #topic carbon/ocata planning
16:22:04 <yamahata> The carbon cycle is short. so we need narrowed plan.
16:22:31 <yamahata> One forcus is to complete unfinished features.
16:23:02 <yamahata> Other focus is to enable those features by default and make them tested with openstack CI.
16:23:28 <yamahata> enable ODL new netvirt and make it tested with openstack ci.
16:23:44 <yamahata> make v2 driver/pseudo agent port binding enabled by default
16:24:13 <yamahata> I tested them a bit.
16:24:14 <mkolesni> i will be trying to fix up networking-odl to make v2 driver the default for ocata
16:24:27 <mkolesni> i found a bug while running tempest with v2
16:24:30 <mkolesni> #link https://review.openstack.org/406968 Add seqnum to dependency checks
16:24:38 <mkolesni> i sent a fix that should fix this
16:24:41 <yamahata> Cool.
16:24:59 <yamahata> #link https://review.openstack.org/#/c/382597/ DO NOT REVIEW/MERGE: test v2 driver
16:25:11 <mkolesni> generally i saw lots of rows stuck in "pending"
16:25:26 <mkolesni> because of circular dependency checks
16:25:32 <yamahata> It showed v2driver is very unstable. it results in failure with poenstack ci.
16:25:45 <mkolesni> yes im following that patch
16:25:53 <yamahata> Cool.
16:26:08 <mkolesni> also trying to recreate the same run locally to get the same results
16:26:44 <mkolesni> currently locally i see much more failures but probably im not configuring devstack/tempest like the gate
16:26:48 <mkolesni> so ill be working on that
16:27:41 <yamahata> Great. right now gate executes tests serially. Make it run in parallel would raise more issue.
16:28:16 <yamahata> I filed https://bugs.launchpad.net/networking-odl/+bug/1646681
16:28:33 <yamahata> I noticed it by looking at the log
16:28:34 <mkolesni> yes i think the parallel makes the circular dependencies issues pop up
16:28:49 <mkolesni> because of the volume of requests in a short time
16:28:58 <yamahata> #link https://bugs.launchpad.net/networking-odl/+bug/1646681  v2 driver: too many skipping row on same resource in short period Edit
16:29:30 <mkolesni> yes
16:29:45 <mkolesni> probably this is the same issue i witnessed
16:29:46 <yamahata> I'm guessing it happens only when there is only one op left in journal. So it's not cirtical.
16:30:21 <yamahata> I haven't dug into it so deeply.
16:30:49 <mkolesni> actually i saw more than 100 rows there
16:31:01 <mkolesni> and the driver just running over them endlessly
16:32:03 <yamahata> that sounds plausible.
16:32:03 <john_a_joyce> Mike - Isaku are aware of anyone using the V2 driver in production that is seeing these issues?
16:32:35 <mkolesni> hi John! no im not aware of any use of this in production yet
16:32:45 <yamahata> No at the moment.
16:32:53 <john_a_joyce> We are using the V2 design on a Cisco specific controller and aren't seeing these issues
16:32:56 <mkolesni> or not production for that instance
16:32:58 <yamahata> ODL new netvirt people are testing it with ODL CI.
16:33:09 <john_a_joyce> it is in a production environment
16:33:24 <yamahata> They say v2 driver is stable with ODL CI.
16:33:30 <mkolesni> i think it happens where theres a lot of requests simultaneously
16:33:40 <yamahata> But I guess it's not production environment.
16:33:44 <john_a_joyce> ok - I will look closely at these bugs and make sure we didn't add anything in our environment
16:33:47 <mkolesni> not sure if this happens frequently in production
16:34:04 <john_a_joyce> generally our driver is behind the v2 driver by a few patches
16:34:37 <john_a_joyce> glad to see the testing is expanding though
16:34:49 <yamahata> In high load case, neutron api process speed can be > journal process speed.
16:35:05 <yamahata> In this case, neutron api process needs to be throttled somehow.
16:35:44 <john_a_joyce> that might be the difference our production environment doesn't pound the neutron API
16:35:52 <mkolesni> probably
16:36:06 <mkolesni> im not sure throttling is a good solution it wont scale
16:36:07 <yamahata> We don't have it right now. journal prcess speed is determined by ODL.
16:36:10 <john_a_joyce> so the API load is moderate at best
16:36:15 <mkolesni> better fix the bugs :)
16:36:49 <yamahata> mkolesni: Yeah bug should be fixed first. Please go ahead :)
16:36:51 <john_a_joyce> yeah throttling usually just moves the bug somewhere else
16:37:29 <mkolesni> as i said sent a patch for the circular dependency stuff we will see if it improves the behavior
16:38:05 <yamahata> #topic patches/bugs
16:38:44 <yamahata> any other bugs/patches?
16:39:05 <mkolesni> none from me
16:39:34 <yamahata> #link http://status.openstack.org/openstack-health/#/g/project/openstack~2Fnetworking-odl
16:40:04 <yamahata> we have priodic job in openstack CI. I noticed periodic job with liberty fails to run
16:40:25 <yamahata> peridodic job with newton is able to run it, but results in test case failure.
16:40:52 <yamahata> Since liberty is about to be end-of-life, it's okay to stop liberty periodic job.
16:41:01 <yamahata> What should we do with mitaka branch? and newton?
16:41:40 <yamahata> Just can we stop those periodic job? or do we want to fix mitaka branch? do we want to add newton periodic job?
16:41:53 <yamahata> #link http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-networking-odl-beryllium-snapshot-liberty
16:42:02 <yamahata> #link http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-networking-odl-boron-snapshot-liberty
16:42:09 <yamahata> #link http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-networking-odl-boron-snapshot-liberty
16:42:32 <mkolesni> whats the point of the periodic job?
16:43:00 <mkolesni> to test against ODL stablke changes once in a while?
16:43:03 <mkolesni> stable
16:43:10 <yamahata> the goal is to test with ODL master head in order to detect breakage.
16:43:21 <yamahata> ODL master + networking-odl stable
16:43:43 <mkolesni> seems reasonable then to have these jobs for the stable branches we support
16:43:46 <yamahata> oh no. ODL stable + networking-odl stable
16:44:13 <mkolesni> unless we have such checks on ODL side already
16:44:45 <yamahata> opendaylight CI has similar. and also OPNFV is testing with stable branch.
16:45:21 <mkolesni> i guess they make more sense to be on ODL side
16:45:28 <yamahata> I'm not monitoring their results, though.
16:45:31 <mkolesni> there is more potential for breakage there
16:46:29 <yamahata> I see. Given the current situation, it would make sense to delete those periodic job.
16:46:47 <yamahata> Anyway if we think it's necessary, we can re-add them again.
16:47:30 <mkolesni> ok
16:47:40 <yamahata> #action yamahata delete odl periodic job from openstack CI.
16:47:54 <yamahata> any other patches/bugs?
16:49:53 <yamahata> BTW Right now my team is trying to fix multinode CI.
16:50:21 <yamahata> I'm wondering to add vagrant file to setup multinode case to the repo because it's difficult to setup its environment locally.
16:50:30 <yamahata> vagrant files for it would help.
16:50:46 <yamahata> anything else or we can move to open mike
16:51:04 <mkolesni> nothing from me
16:51:11 <yamahata> #topic open mike
16:51:24 <yamahata> let me discuss on stable branch maintenance.
16:51:57 <yamahata> In recent discussion on netvirt mailing list, there seems requirement for stable/newton branch to have backports.
16:52:25 <yamahata> mkolesni: do you have any interest?
16:52:25 <mkolesni> dont think im on that list
16:52:30 <mkolesni> what do they want backported?
16:52:58 <yamahata> #link https://lists.opendaylight.org/mailman/listinfo/netvirt-dev
16:53:34 <yamahata> At the moment, they don't have concrete ones. but furture maintenance.
16:53:51 <yamahata> e.g. bug fixes for v2 driver, pseudo agent port binding
16:54:15 <yamahata> As we'll migrate to new netvirt, netwokring-odl may have more bug fixes realted to it.
16:54:26 <mkolesni> well lets see what pops up and tackle it as it arrives
16:54:38 <mkolesni> i thought there something concrete being discussed
16:55:04 <yamahata> ok.
16:55:47 <mkolesni> you have something in mind?
16:55:47 <yamahata> any other topic to discuss?
16:56:00 <yamahata> For now no.
16:56:06 <mkolesni> ok
16:56:29 <john_a_joyce> nothing from me
16:56:46 <yamahata> next week Dec 12, we will have meeting.
16:57:04 <yamahata> On Dec 19 I can host it.
16:57:14 <yamahata> We will skip Dec 26 and Jan 2.
16:57:20 <mkolesni> ok
16:57:49 <yamahata> I'll give you 3min back
16:57:53 <yamahata> thank you everyone
16:57:55 <mkolesni> ok :)
16:57:59 <mkolesni> thanks Isaku
16:58:07 <yamahata> #topic cookies
16:58:11 <yamahata> #endmeeting