15:02:57 <yamahata> #startmeeting neutron_northbound
15:02:57 <odl_meetbot> Meeting started Mon Aug 21 15:02:57 2017 UTC.  The chair is yamahata. Information about MeetBot at http://ci.openstack.org/meetbot.html.
15:02:57 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:57 <odl_meetbot> The meeting name has been set to 'neutron_northbound'
15:03:13 <yamahata> #chair mpeterson rajivk manjeets
15:03:13 <odl_meetbot> Current chairs: manjeets mpeterson rajivk yamahata
15:03:19 <yamahata> #topic agenda bashing and roll call
15:03:23 <yamahata> #info yamahata
15:03:28 <mpeterson> #info mpeterson
15:03:32 <rajivk> #info rajivk
15:03:41 <manjeets_> #info manjeets
15:03:43 <yamahata> #link https://wiki.opendaylight.org/view/NeutronNorthbound:Meetings
15:03:53 <yamahata> today main topics will be patch/bug
15:03:59 <yamahata> any other topics?
15:04:04 <mpeterson> CI tests
15:04:26 <mpeterson> and I guess full sync should be talked about
15:04:50 <yamahata> yeah, CI and fullsync are biggest issues.
15:05:33 <yamahata> anything else?
15:06:07 <yamahata> okay move on.
15:06:08 <yamahata> #topic Announcements
15:06:26 <yamahata> now Pike RC phase and we're focusing on bug fixes and stabilizing it.
15:06:50 <yamahata> On odl side, it's also rc phase. stable/nitrogen branch has been cut.
15:07:01 <yamahata> openstack PTG is approaching
15:07:16 <yamahata> the topics are gathered at https://etherpad.openstack.org/p/neutron-queens-ptg
15:07:30 <yamahata> ODL DDF is planned in santa clara on october
15:07:56 <yamahata> #link https://wiki.opendaylight.org/view/Events:Oxygen_Dev_Forum
15:08:01 <yamahata> any other announcement?
15:08:52 <yamahata> okay move on
15:08:53 <yamahata> #topic action items from last meeting
15:09:04 <yamahata> there is no action items except bug/patch.
15:09:10 <yamahata> #topic Pike/Nitrogen planning
15:09:23 <yamahata> #topic patches/bugs
15:09:27 <yamahata> Okay we have many issues.
15:09:43 <yamahata> #link https://bugs.launchpad.net/neutron/+bugs?field.searchtext=&search=Search&field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&fiel
15:09:43 <yamahata> d.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&orderby=-id&start=0 netwokring-odl bugs
15:10:02 <yamahata> #link https://review.openstack.org/#/q/project:openstack/networking-odl+status:open pending patches
15:10:14 <manjeets_> tiny url yamahata ?
15:10:22 <yamahata> oops
15:11:00 <yamahata> #link https://tinyurl.com/ybq9zm4l networking-odl bugs
15:11:23 <manjeets_> thanks
15:11:38 <yamahata> We have several bugs and some of then need to be addressed before pike release.
15:11:51 <yamahata> One is full sync and otherone is CI related.
15:12:15 <yamahata> Now we know fullsync is heavily broken and several patches have been proposed.
15:12:30 <manjeets_> yamahata, what about graceful http error handling, should that go as well or can wait ?
15:12:50 <yamahata> It causes many error stack traces. it should be fixed for Pike.
15:13:09 <rajivk> yamahata, if someone can test it, will be good. I am testing full_sync for now.
15:13:28 <yamahata> manjeets: can you test it as reviewer?
15:13:50 <manjeets_> I reviewed it already looks good to me know
15:13:54 <rajivk> All the scenario can be checked once by creating scenarios and then let's merge it.
15:14:02 <yamahata> Oh you've already gave it +1.
15:14:20 <rajivk> it is end of release, i don't want to commit mistake again.
15:15:33 <yamahata> Okay, For now the sympton is annoying error stack trace, we can delay the patch and backport it after testing.
15:16:07 <rajivk> yamahata, okay
15:16:11 <manjeets_> yamahata, +
15:16:25 <yamahata> next is fullsync
15:16:42 <yamahata> Now I've summarised bugs at https://wiki.opendaylight.org/view/NeutronNorthbound:Meetings#Bugs
15:17:14 <yamahata> several patches can be merged for fullsync after review.
15:17:38 <yamahata> on the other hand (unit) test coverage of fullsync is bad.
15:17:43 <yamahata> any volunteer?
15:18:04 <mpeterson> IMHO, first they should be added as bugfixing is happening
15:18:25 <mpeterson> and then a more planned unit test is needed
15:18:25 <rajivk> yamahata, i will work on them.
15:18:34 <yamahata> #link https://review.openstack.org/#/c/494330/
15:18:43 <yamahata> #link https://review.openstack.org/#/c/494961/
15:18:51 <yamahata> #link https://review.openstack.org/#/c/494663/
15:19:30 <yamahata> 494330 looks ready to merge.
15:19:47 <yamahata> 494961 needs to be completed.
15:20:09 <yamahata> 494663 looks bug work around. the true fix should be in Neutron.
15:20:17 <yamahata> any thoughts?
15:20:25 <yamahata> comments?
15:20:34 <rajivk> do we want to fix 494663?
15:20:52 <rajivk> or ask neutron guys fix it on high priority.
15:21:08 <mpeterson> yamahata: I'm worried that some of them seem big and not thoroughly tested thus increasing the tech debt
15:21:28 <yamahata> If https://review.openstack.org/#/c/494953 is merged into pike, we don't need 494663.
15:21:37 <mpeterson> plus there seems to be a trend of more patching than well thought code
15:21:45 <yamahata> mpeterson: do you mean 494961?
15:22:09 <rajivk> yamahata, it is already merged.
15:22:10 <mpeterson> yamahata: it's a trend that I saw in all full sync related patches
15:22:13 <manjeets_> yamahata, https://review.openstack.org/#/c/494953 neutron change merged
15:22:23 <mpeterson> including that one
15:22:40 <yamahata> 494953 is merged for master. not for stable/pike.
15:22:54 <manjeets_> ohk may be back port
15:23:36 <yamahata> mpeterson: I see. do you have any approaches/cleanup on fullsync?
15:23:55 <manjeets_> just issued a cherrypick to stable/pike https://review.openstack.org/#/c/495904/
15:24:24 <yamahata> For Pike, it's too late for big change. So (dirty) patching might be embraced. For queens, we should have clean approach.
15:25:03 <mpeterson> yamahata: well, a question that rises to mind is, does it have to be stabilized before pike or it can wait more?
15:26:16 <yamahata> Given the timeline, I'm inclined to give up to fix it for Pike 1st release, and later to have backport.
15:26:23 <mpeterson> yamahata: if the latter, then like you said we need the tests and those tests should be designed according to criticality
15:26:32 <yamahata> we can't change release schedule to align with openstack release.
15:27:03 <yamahata> Yeah, we should have tests.
15:27:30 <mpeterson> yamahata: then I guess will have to allow some of the patches to be integrated but with the idea of reverting them on a schedule and give a more thoughtful approach
15:28:25 <mpeterson> mpeterson: because if we incorporate different criterias in different parts then it will be a nightmare in the future
15:29:09 <mpeterson> *yamahata
15:29:44 <yamahata> mpeterson: that sounds reasonable.
15:30:15 <manjeets_> mpeterson, ++
15:30:34 <yamahata> fullsync has been experimental, and it seems like people starts to test it seriously.
15:31:15 <mpeterson> I think we should have perhaps decide on a place to discuss this, while bug squashing keeps going
15:31:40 <mpeterson> or you think this meeting should be the place to do it so?
15:32:05 <yamahata> Yeah, surely we can discuss it here.
15:33:09 <yamahata> can you please update meeting page?
15:33:25 <yamahata> https://wiki.opendaylight.org/view/NeutronNorthbound:Meetings
15:33:31 <mpeterson> sure
15:33:33 <yamahata> so that we won't miss it.
15:33:38 <mpeterson> of course
15:34:13 <yamahata> anything else on fullsync?
15:34:38 <mpeterson> not here
15:35:25 <yamahata> okay next one is CI.
15:35:39 <yamahata> Now tempest jobs are all non-voting.
15:35:45 <yamahata> we should have some voting.
15:35:52 <manjeets_> yamahata, any response from netvirt folks ?
15:35:57 <manjeets_> on ssh issues ?
15:36:00 <yamahata> manjeets:  no.
15:36:13 <yamahata> probably we should actively engage them.
15:36:24 <yamahata> Just filing bug report isn't enough.
15:36:32 <yamahata> v1driver job is only for comparison.
15:36:46 <manjeets_> I thought someone was going to ask them in last tuesday's meeting
15:36:58 <yamahata> carbon + v2driver and/or nitrogen + v2driver would be candidate to make voting.
15:37:28 <yamahata> Yeah. At least I'm attending it and planning to ask it tomorrow.
15:37:38 <yamahata> But more attendee would help.
15:38:01 <manjeets_> ok thanks !
15:38:24 <yamahata> At least we should ask netvirt folks on which version we should focus. carbon or nitrogen.
15:38:33 <yamahata> longer term master branch...
15:39:29 <yamahata> Interestingly we have success with carbon jobs, but nitrogen tempest jobs always fail so far.
15:39:41 <yamahata> sometimes success with carbon.
15:40:07 <manjeets_> is it ready to make voting ?
15:40:17 <yamahata> Not at all.
15:40:17 <manjeets_> or some failures need to fixed first
15:40:24 <manjeets_> ohk
15:40:37 <yamahata> #link http://grafana.openstack.org/dashboard/db/networking-odl-failure-rate
15:40:45 <mpeterson> and more worrisome, python27 started failing randomly as well
15:40:47 <yamahata> the failure rate more than 50%
15:41:22 <yamahata> mpeterson: recently?
15:41:35 <mpeterson> yamahata: I started noticing it since last week
15:41:53 <mpeterson> yamahata: been looking sporadically at the logs to see if I see a pattern
15:42:09 <mpeterson> yamahata: I have some ideas on what to look for already
15:42:28 <yamahata> mpeterson: cool. Please go fix it.
15:43:17 <yamahata> anthing else to share?
15:43:34 <mpeterson> yamahata: two main things I noticed are: 1) the test run can't finish the thread even though it finished testing; 2) we have a couple of tracebacks that are not making the test fail
15:43:55 * mpeterson has another thing!
15:44:17 <mpeterson> yamahata: we should perhaps ask infra to recollect logs even when timeouts occur
15:44:36 <yamahata> mpeterson: can you volunteer to do that?
15:44:36 <mpeterson> yamahata: as I understand that it's not on our sphere of influence, right?
15:45:17 <mpeterson> yamahata: I can volunteer to at least update you next week on what I did, if anything :)
15:45:18 <yamahata> we can ask them to persuade that getting logs even with timeout is critical...
15:45:24 <yamahata> Great.
15:45:35 <mpeterson> yamahata: with all the intention to try to do something
15:46:03 <yamahata> I'm also trying to get logs with functional/fullstack tests. I also noticed many stacktraces.
15:46:18 <yamahata> So far I haven't assessed them, though.
15:46:27 <mpeterson> another thing, but I can't volunteer for too much more
15:46:58 <mpeterson> We should have a task in the CI to collect std{out,err} and unhandled exceptions and perhaps create a bug or something
15:47:00 <yamahata> we can't do all the thing.
15:47:19 <yamahata> CI? doesn't tox collect them?
15:47:38 <mpeterson> it collects them, but if it reports success on the CI taks no one will go to read the logs
15:47:42 <yamahata> Maybe some logs might be missed to upload...
15:48:11 <yamahata> mpeterson: can you file a bug report to track it?
15:48:17 <yamahata> at least we don't want to forget it.
15:48:27 <mpeterson> yamahata: no, I'm not talking about missing logs. They are there. However, since the task succeeds no one ever looks at them
15:48:44 <mpeterson> even if you run tox locally, for example you'll get deprecation warnings
15:48:53 <mpeterson> and that's the most basic example
15:49:11 <yamahata> Ah you mean assessing exceptions in logs and file them, right?
15:49:15 <mpeterson> and we do have deprecations warnings with Pike labeled as the version to be removed
15:49:20 <mpeterson> yamahata: correct
15:49:56 <yamahata> that surely makes sense and we need volunteer.
15:51:09 <yamahata> I filed it for tracking. https://bugs.launchpad.net/neutron/+bug/1712109
15:51:26 <yamahata> we have 9mins left.
15:51:34 <yamahata> anything else to discuss?
15:51:38 <rajivk> yamahata, i will look into it, after full sync.
15:51:43 <manjeets_> yamahata, i have one thing
15:52:01 <yamahata> rajivk: great. I'm aware that you're also looking at bugs.
15:52:07 <mpeterson> yamahata, rajivk: great!
15:52:08 <yamahata> manjeets: please go ahead.
15:52:11 <manjeets_> added a doc to provide reference deployment versions
15:52:14 <manjeets_> https://review.openstack.org/#/c/493966/
15:52:30 <manjeets_> reviews needed so that I make further progress
15:52:41 <rajivk> yamahata, one more thing
15:52:57 <rajivk> https://bugs.launchpad.net/networking-odl/+bug/1711286
15:52:57 <yamahata> #action everyone review https://review.openstack.org/#/c/493966/
15:53:21 <yamahata> rajivk: please go ahead.
15:53:32 <rajivk> I notice it temporary. I don't know, what is causing host config to change.
15:54:04 <rajivk> Sometimes, it returns host config and work fine and sometimes fails.
15:54:13 <rajivk> What could be the possible reason?
15:54:17 <yamahata> what failure are you seeing?
15:54:25 <yamahata> I commented on your bug report.
15:54:38 <rajivk> yeah, i went through it.
15:54:57 <rajivk> Do you think, we should flush host config on instllation time.
15:55:17 <rajivk> SO any previous  configuration is lost
15:55:33 <yamahata> No. basically hostconfig has compatibility.
15:55:35 <rajivk> I used devstack and the same local.conf work on other machine fine.
15:55:42 <yamahata> And recently It doesn't have format change.
15:56:15 <rajivk> yamahata, i will look into it. and share complete report.
15:56:33 <rajivk> that's all from my side.
15:56:34 <yamahata> The recent change is to retrieve it if agentdb is missing when portbinding
15:56:58 <yamahata> rajivk: thanks.
15:57:04 <yamahata> #topic open mike
15:57:07 <yamahata> anything else?
15:57:53 * manjeets_ wents out to watch solar eclipse
15:57:56 <yamahata> it seems okay to close.
15:58:00 <yamahata> thank you everyone
15:58:04 <manjeets_> thanks
15:58:08 <yamahata> #topic cookies
15:58:10 <rajivk> thanks
15:58:13 <yamahata> #endmeeting