15:04:33 <yamahata> #startmeeting neutron_northbound
15:04:33 <odl_meetbot> Meeting started Wed Aug  3 15:04:33 2016 UTC.  The chair is yamahata. Information about MeetBot at http://ci.openstack.org/meetbot.html.
15:04:33 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:04:33 <odl_meetbot> The meeting name has been set to 'neutron_northbound'
15:04:41 <yamahata> #chair mkolesni
15:04:41 <odl_meetbot> Current chairs: mkolesni yamahata
15:04:51 <yamahata> #topic agenda bashing and roll call
15:04:54 <yamahata> #info yamahata
15:05:06 <yamahata> #link https://wiki.opendaylight.org/view/NeutronNorthbound:Meetings meeting agenda page
15:05:09 <mkolesni> #info mkolesni
15:05:29 <yamahata> any additional agenda?
15:06:18 <yamahata> seems nothing else. move on
15:06:21 <yamahata> #topic Announcements
15:06:30 <yamahata> opendaylight offset 2 M5 is tomorrow.
15:06:42 <yamahata> and new branch will be cut.
15:07:09 <yamahata> Next wednesday I would be absent.
15:07:27 <yamahata> Unless anyone else chair the meeting, it would be skipped.
15:07:45 <yamahata> #action yamahata announce Isaku's absent next week on the mailig list
15:07:51 <mkolesni> I can chair it if there will be interest
15:08:01 <yamahata> Cool.
15:08:03 <barak> Hi Everyone
15:08:10 <yamahata> Hi barak
15:08:29 <barak> Not sure if that is the correct place to raise such issues...
15:08:34 <yamahata> any other announcement?
15:09:32 <yamahata> barak: this channel is for opendaylight and openstack neutron related tech issues.
15:09:52 <yamahata> seems no other announcement.
15:09:54 <yamahata> #topic action items from last meeting
15:10:05 <barak> We think of using the v2 with mitaka. I noticed some problems
15:10:10 <yamahata> I created a page for new migration https://wiki.opendaylight.org/view/NeutronNorthbound:MigrationToNewFeatures
15:10:35 <yamahata> barak: can you wait a bit until discussion bug, issues?
15:10:43 <yamahata> It's soon.
15:10:59 <yamahata> #link https://wiki.opendaylight.org/view/NeutronNorthbound:MigrationToNewFeatures summary page for feature migration
15:11:15 <yamahata> and vithal announce openstack summit voting
15:11:23 <barak> sure
15:11:44 <yamahata> #topic patches/bugs
15:11:50 <yamahata> barak: now you're on stage
15:12:11 <yamahata> I
15:12:31 <yamahata> I've also started to test v2 driver to hit issues.
15:12:39 <barak> thanks. I have loaded v2 with mitaka, and there seems to be a missing talble on the database
15:12:53 <barak> the database is used for sync
15:12:58 <mkolesni> v2 is not functioning in mitaka
15:13:17 <mkolesni> its mostly functioning now on the master branch, except few bugs here and there
15:13:30 <mkolesni> can you try the master branch?
15:13:48 <yamahata> neutron master + networking-odl master
15:13:50 <barak> I have some other dependencies ...
15:14:38 <barak> Are there any additional issues except from the table?
15:15:28 <mkolesni> its mostly not ready there and had issues not fixed in mitaka
15:15:55 <mkolesni> i suggest not try it in mitaka branch unless maybe for POC
15:18:21 <yamahata> I suppose almost all developers are testing v2 driver with master branch.
15:19:57 <barak> Yes, I know it was meant for next version. But codewise, any additional issues you are aware of, other than the table? We have run tempest with many successfull.
15:20:24 <mkolesni> there are race conditions there
15:20:32 <yamahata> Ah, there are many interesting issues. :-) I'd like to discuss them today.
15:21:01 <yamahata> I have two issues today.
15:21:07 <yamahata> One is dbdeadlock error.
15:21:27 <yamahata> I'm observing dbdeadlock error in precommit with tempest.
15:21:48 <yamahata> I filed RFE for neutron
15:21:58 <yamahata> #link https://bugs.launchpad.net/neutron/+bug/1609184 [RFE] ML2: Allow retry on db error by precommit
15:22:10 <yamahata> #link https://review.openstack.org/#/c/350315/
15:22:10 <yamahata> ml2: allow retry on retriabable db error by precommit
15:22:37 <mkolesni> this isnt supported there even though we added the decorator?
15:22:45 <yamahata> I needed the patch to reliably pass tempest.api.network.*
15:23:21 <yamahata> The problem is that ML2 swallows the exception and reports it as error to user.
15:23:28 <mkolesni> :/
15:23:37 <yamahata> and retry needs to be done at top level of transaction.
15:24:00 <yamahata> So a patch to neutron is necessary.
15:24:40 <yamahata> Another issue is journal syncer thread.
15:24:54 <mkolesni> ok makes sense so we need to document for now that for galera multi-writer it isn't working
15:25:35 <yamahata> Unfortunately it happens with normal backend with all-in-one setup.
15:26:14 <mkolesni> so we need to investigate further
15:26:28 <yamahata> I have already a patch. https://review.openstack.org/#/c/350315/
15:27:09 <yamahata> I will raise it at ML2 irc meething which starts after this irc meeting.
15:27:19 <mkolesni> ok
15:27:48 <mkolesni> if this doesnt have anything to do with galera it would be better to understand why theres a deadlock
15:28:13 <mkolesni> if its normal backend then its a real deadlock probably
15:28:42 <yamahata> It happens only when tempest.api.network run with parallel option enabled.
15:29:39 <yamahata> with subnet CRUD, it touches other resource like port, ip address. I suspect it gains table lock in some order.
15:30:08 <yamahata> It may cause dbdeadlock.
15:30:21 <mkolesni> do you have a bug for that?
15:30:35 <mkolesni> not the ML2 just one documenting the deadlock
15:30:45 <yamahata> #link https://bugs.launchpad.net/networking-odl/+bug/1609149
15:31:15 <mkolesni> ill try take a look at it
15:31:16 <yamahata> I uploaded a log there.
15:32:12 <yamahata> On the other hand, syncer thread reads tables reverse order. journal -> resource.
15:32:37 <yamahata> for dependency validation
15:33:12 <mkolesni> probably happens when the thread is fired by the timer
15:33:23 <yamahata> #action mkolesni look into dbdeadlock bug report
15:33:32 <mkolesni> otherwise it's triggered by post commit so it shouldnt cause deadlock
15:33:49 <yamahata> yeah, that's same to my suspect.
15:34:11 <mkolesni> ok thanks for the info ill take that into consideration
15:34:30 <yamahata> next issue I observed is
15:34:45 <yamahata> journal syncer thread fails to process entries.
15:35:06 <yamahata> It stuck with dependency validation for some reasons.
15:35:26 <yamahata> I haven't looked into it deeply. I'll do today.
15:35:32 <mkolesni> so live lock?
15:35:39 <yamahata> Not sure.
15:35:55 <yamahata> It can be easily reproducible by runnint tempest.api.network.*
15:36:08 <mkolesni> ok
15:36:33 <yamahata> #link https://bugs.launchpad.net/networking-odl/+bug/1609359 devstack regression: neutron-odl-ovs-hostconfig doesnt exist on compute hosts, prevents devstack from loading
15:36:40 <yamahata> There is a regression. Sorry for that.
15:36:50 <yamahata> My team will look into it.
15:37:18 <mkolesni> ok
15:37:32 <yamahata> I also uploaded several patches to address issues I observed.
15:37:49 <yamahata> Some may be too early to upload, though.
15:38:37 <mkolesni> can you please mark workflow -1 whats not ready?
15:38:48 <yamahata> sure
15:39:00 <yamahata> #action yamahata mark workflow -1 to what's not ready.
15:40:13 <yamahata> that's all from me to share this week.
15:40:28 <yamahata> any other bugs/patches?
15:40:42 <mkolesni> nothing from me
15:41:13 <yamahata> #topic Boron planning
15:41:23 <yamahata> The remaining thing is documentation
15:41:31 <yamahata> #action yamahata write/update documentation
15:41:51 <yamahata> and it's time to start planingfor carbon
15:41:56 <yamahata> #topic open mike
15:42:04 <yamahata> anything else to discuss?
15:42:13 <mkolesni> nothing from me
15:42:50 <yamahata> can we skip bluejeans meeting?
15:42:57 <mkolesni> sure
15:46:47 <yamahata> okay thank you for everyone
15:47:01 <yamahata> #topic cookies
15:47:08 <yamahata> #endmeeting