15:01:01 <yamahata_> #startmeeting neutron_northbound
15:01:01 <odl_meetbot> Meeting started Mon Jun 19 15:01:01 2017 UTC.  The chair is yamahata_. Information about MeetBot at http://ci.openstack.org/meetbot.html.
15:01:01 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:01 <odl_meetbot> The meeting name has been set to 'neutron_northbound'
15:01:08 <yamahata_> #chair rajivk
15:01:08 <odl_meetbot> Current chairs: rajivk yamahata_
15:01:16 <yamahata_> #topic agenda bashing and roll call
15:01:25 <yamahata_> #info yamahata
15:01:34 <rajivk> #info rajivk
15:01:40 <yamahata_> any topics in addition to usual ones?
15:01:59 <rajivk> not
15:02:00 <yamahata_> right now, openstack CI is heavily broken. so I'd like to share it later.
15:02:28 <yamahata_> #topic Announcements
15:02:49 <yamahata_> ODL karaf4 migration is on-going.
15:03:34 <yamahata_> #link https://wiki.opendaylight.org/view/Karaf_4_migration
15:04:13 <yamahata_> Within 2 weeks, nitrogen branch may be broken heavily. Especially all the projects are kicked out from karaf distribution.
15:04:20 <yamahata_> And then will be added.
15:05:00 <yamahata_> also karaf feature start up would be issue as it's known.
15:05:12 <mkolesni> hi
15:05:15 <yamahata_> With karaf4 migration, bootFeature may be dropped.
15:05:18 <yamahata_> mkolesni: hi.
15:05:18 <mkolesni> #info mkolesni
15:05:27 <mkolesni> sorry had some traffic on the way home
15:06:45 <yamahata_> The next thing is, right now networking-odl is heavily broken.
15:06:58 <yamahata_> The issue is already tracked down.
15:07:08 <yamahata_> devstack change and neutron security group ovo.
15:07:36 <yamahata_> several patches are on the way. So please help to make them merged
15:08:08 <yamahata_> #link https://bugs.launchpad.net/networking-midonet/+bug/1698129 devstack fails trying to create systemd user unit file for q-agt
15:08:26 <yamahata_> This is the devstack issue
15:08:56 <yamahata_> the another one is, neutron security group ovo broke several decomposed modules.
15:09:26 <yamahata_> The patch for neutron, https://review.openstack.org/#/c/448420/ was already merged.
15:09:32 <manjeets> hi
15:10:06 <yamahata_> So the patches for networking-odl can be found at https://review.openstack.org/#/q/topic:security-group-ovo
15:10:25 <yamahata_> Finally, they need to be squashed into single patch to make CI pass.
15:10:28 <yamahata_> manjeets: hi
15:10:51 <yamahata_> That's the current situation and what I'm aware of.
15:11:00 <yamahata_> any thing else to announce?
15:12:12 <yamahata_> seems nothing, move on.
15:12:15 <yamahata_> #topic action items from last meeting
15:12:38 <mkolesni> 961735
15:12:40 <yamahata_> no action items except patch review.
15:12:45 <mkolesni> sry ignore
15:13:09 <yamahata_> #topic patches/bugs
15:13:22 <yamahata_> any patches/bugs?
15:13:25 <yamahata_> mkolesni: you're on stage
15:13:27 <mkolesni> yes
15:14:06 <yamahata_> wrong number?
15:14:15 <mkolesni> ive been running some perf testing on https://review.openstack.org/#/q/status:open+project:openstack/networking-odl+branch:master+topic:bp/dep-validations-on-create
15:14:23 <mkolesni> rally mostly
15:15:05 <mkolesni> i didnt see any improvement or any significant negative impact on performance
15:15:25 <mkolesni> the most improvement i saw was when ODL was down
15:15:59 <mkolesni> now i have pretty basic h/w so not sure if cranking up the thread count might lead to different results
15:16:14 <mkolesni> so basically we need to decide if we want this or not
15:16:51 <mkolesni> the main thing about the change of dependency calculation is that it prevents coders from introducing certain bugs such as cyclic dependency
15:17:01 <yamahata_> Are those performance improvement, right?
15:17:18 <mkolesni> or for example a bug where older entry depends on newer entry
15:17:39 <mkolesni> the main goal was always the code improvement not performance improvement
15:17:42 <manjeets> we can do a review if it make code simpler, I guess it be worth keeping it to prevent future bugs
15:18:29 <mkolesni> sure feel free
15:18:40 <yamahata_> 467060 looks like breaking the core logic.
15:19:02 <yamahata_> If we'd like to avoid lock for update, for example, we can do update where limit 1 etc.
15:19:27 <mkolesni> no thats one is needed because otherwise theres deadlocks because of cross dependency not being handled well by mysql
15:19:35 <yamahata_> Then we can pick journal entry with single sql statement.
15:19:59 <mkolesni> its not breaking anything, how do you arrive at that conclusion?
15:20:09 <yamahata_> cross dependency? can you please elaborate it?
15:21:00 <mkolesni> the change 453581 changes so that not any entry is selected but just latest that doesnt have unprocessed dependencies
15:21:41 <mkolesni> so upon insert into dependency table it will hold a lock already over the journal table
15:22:00 <mkolesni> obviously since the journal entry is first created and then the dependency table rows are created
15:22:32 <mkolesni> but the select for update lock in the opposite direction, first the dependency table and then the journal table is locked
15:22:44 <mkolesni> so obviously opposite order here results in deadlock
15:22:56 <mkolesni> i know of no way to fix the order of locking in the select
15:23:10 <mkolesni> and in the inserting transaction the journal table has to be created beforehand
15:23:10 <yamahata_> 453851? I suppose typo. it's for openstack-infra/zuul
15:23:42 <mkolesni> <mkolesni> the change 453581 ...
15:23:47 <mkolesni> no typo
15:23:55 <mkolesni> https://review.openstack.org/453581
15:23:59 <mkolesni> full link
15:24:23 <yamahata_> ho sorry. my typo locally
15:24:27 <mkolesni> so the only way to avoid tons of deadlocks that i know is to change to optimistic locking
15:24:37 <mkolesni> unless someone has a better idea
15:25:03 <mkolesni> i noticed that on a rally gate run the number of deadlocks can reach even 1000
15:25:48 <mkolesni> problem is mysql usually will restart the transaction with the insert and not the other one, even thought the inserting one is doing more work
15:25:59 <mkolesni> otherwise we could leave with a deadlock
15:26:48 <mkolesni> but when a port operation takes ~10 seconds end to end, a deadlock causes retry which will take this same time so it might at works case take 5x time to retry (until max retries is eached)
15:27:54 <mkolesni> hence the optimistic locking patch
15:28:34 <yamahata_> Is your rally run with the patch of https://review.openstack.org/#/c/453581/ ?
15:28:50 <yamahata_> If so, I suppose I understand your description.
15:29:27 <mkolesni> yes
15:29:39 <mkolesni> its build on top of each other
15:30:34 <yamahata_> gotcha
15:32:13 <mkolesni> so we need to decide on this change as a whole
15:32:46 <yamahata_> My concern is 453581.
15:33:10 <yamahata_> What about if two resources with dependency are updated simultaneously?
15:33:11 <mkolesni> i think since theres no performance impact then its better than existing code for dependency calculations which is more bug prone
15:33:35 <yamahata_> Doesn't it miss depenency? or don't we need to calculate dependency atomically somehow?
15:33:42 <mkolesni> this was discussed previously and we arrived at the conclusion its impossible
15:33:52 <mkolesni> see YAMAMOTOs comment
15:34:00 <mkolesni> on this issue
15:34:07 <yamahata_> It
15:34:12 <yamahata_> It's with same resource.
15:34:18 <mkolesni> yes
15:34:23 <mkolesni> same resource
15:34:25 <yamahata_> How about different type. e.g. security group and security group rule?
15:34:37 <yamahata_> or whatever two kind of resources?
15:35:05 <mkolesni> so whats the problem there?
15:35:27 <mkolesni> if you have a concrete concern please explain whats the possible bug there
15:35:50 <yamahata_> Suppose network update and port update.
15:36:05 <mkolesni> ok what will exactly cause a bug? what payload the update carries?
15:36:06 <yamahata_> network and port are only example. generally two type of resource with dependency.
15:36:35 <yamahata_> dependency calculation for network and port is done without other update.
15:37:06 <mkolesni> so what? whats the exact payload of update that causes a problem?
15:37:08 <yamahata_> then the dependency table will be inserted at the same time.
15:37:19 <yamahata_> so the late commer will miss the dependency.
15:38:09 <mkolesni> again not sure what the exact problem with payload there?
15:38:18 <yamahata_> you don't care? then why do we have depdenency validation?
15:38:22 <mkolesni> what is the network update?
15:38:39 <mkolesni> the scenario youre describing could happen today as well
15:38:53 <mkolesni> so im not sure whats exactly your concern
15:39:28 <mkolesni> the patch doesnt aim to solve every single problem, just make the dependency calculations mechanism more resillient
15:39:56 <mkolesni> it cant solve any problem in networking odl dependencies since thats not the aim
15:40:09 <yamahata_> The patch introduces new issue.
15:40:17 <mkolesni> again no it doesnt
15:40:23 <yamahata_> If we don't care inter-resource dependency,
15:40:23 <mkolesni> this same issue can happen today
15:40:32 <mkolesni> this can happen with todays code
15:40:39 <yamahata_> okay, we can drop whole dependency validation logic. we can have only same resource check.
15:40:43 <mkolesni> the winner of the race will be independent
15:40:55 <mkolesni> im not sure what you want
15:41:01 <yamahata_> Is it the direction you'd like to go for?
15:41:08 <mkolesni> this problem exists today and im not trying to solve it
15:41:27 <mkolesni> no the direction is to improve the code without introducing new issues
15:41:40 <mkolesni> this is not a new issue youre describing
15:41:48 <mkolesni> and also not a very interesting one
15:41:56 <mkolesni> but thats besides the point
15:42:29 <mkolesni> but by your logic maybe we should drop the entire V2 driver and go back to V1
15:42:43 <mkolesni> since V2 driver obviously has issues not present in V1 driver
15:42:51 <mkolesni> is that your direction?
15:43:06 <yamahata_> No. I'm guessing direction you'd like to go for?
15:43:27 <mkolesni> i feel this project has too much bike shedding and not enough real work done
15:43:52 <mkolesni> for instance you approved a patch with numerous issues
15:44:09 <mkolesni> so now we had to revert and re-work it to iron out the issues
15:44:44 <yamahata_> I'm very glad that you jumped in to address that patch.
15:44:50 <mkolesni> so please lets try to avoid bike shedding and get some work done, we dont have much until pike-3 which is feature freeze
15:45:47 <yamahata_> Right now, we would need more reviewer to cover as a whole.
15:45:58 <yamahata_> not only the area interested to contributor.
15:46:09 <mkolesni> yes that would also help with project health
15:47:34 <mkolesni> but less bike shedding and focusing on unreal stuff and instead making real iterative progress would also be very beneficial
15:48:48 <yamahata_> I agree with less bike shedding.
15:49:51 <mkolesni> great so lets try to make some real progress
15:50:10 <mkolesni> pike 3 is quickly approaching and everyone has other work to take care of
15:50:52 <mkolesni> ok please if you find real issues post it on review
15:51:03 <yamahata_> So far CI isn't in good shape. especially ssh via floating ip doesn't work with tempest.
15:51:06 <mkolesni> but please dont post issues that already exist in current system
15:51:33 <mkolesni> i can try to take a look at it but this week im swamped and also i have to review rajivk's patch
15:51:34 <yamahata_> So we've disabled related tests. They needs fixed and also several combination isn't working.
15:51:57 <mkolesni> also next week ill be at a company summit thing so i will be of limited availability
15:52:11 <mkolesni> ok
15:52:26 <yamahata_> Oh i see. Enjoy the event!
15:52:52 <mkolesni> thanks
15:52:58 <mkolesni> :)
15:53:12 <yamahata_> except the recent breakage, the situation is't good. we need to address them for neutron stadium.
15:53:21 <yamahata_> okay, we're running out of time.
15:53:26 <yamahata_> anything else to discuss?
15:53:31 <mkolesni> nothing from me
15:53:33 <yamahata_> #topic open mike
15:53:43 <yamahata_> #link http://grafana.openstack.org/dashboard/db/networking-odl-failure-rate
15:53:52 <manjeets> yamahata_, a suggestion
15:54:17 <yamahata_> manjeets: sure. what's suggestion?
15:54:23 <manjeets> if everyone agrees we can have slot for discussing new features within this meeting
15:54:34 <manjeets> like dhcp-port for odl is coming
15:55:06 <mkolesni> we can do that but please keep in mind that after pike-3 we should be stabilizing the code and fixing bugs
15:55:19 <manjeets> like we discuss bugs so we'll have a overview in this meeting what new things are coming will help on prioritizing reviews
15:55:32 <mkolesni> especially since we switched to release with milestones work mode
15:55:50 <manjeets> mkolesni, sure !! I agree
15:56:34 <mkolesni> ok so we can discuss it in the bugs slot or before that
15:57:13 <rajivk> i want to discuss about db locking patch.
15:57:14 <yamahata_> we can give it a try to experiment.
15:57:36 <yamahata_> rajivk: please go ahead.
15:57:44 <yamahata_> mkolesni: do you have time?
15:57:51 <mkolesni> i have to go in a couple of minutes
15:58:00 <rajivk> Currently i have made periodic class singelton
15:58:15 <rajivk> we will not be able to configure all the tasks time interval
15:58:21 <mkolesni> rajivk, can we do it earlier tomorrow or do you need yamahata_ for this?
15:58:38 <rajivk> we can discuss it tomorrow as well
15:58:41 <rajivk> that's fine
15:58:48 <mkolesni> ok cool
15:59:01 <mkolesni> i have an idea but lets discuss tomorrow i really gtg
15:59:14 <yamahata_> anything else last minutes?
15:59:19 <rajivk> yes
15:59:38 <rajivk> yamahata, i would like db locking and threadpool patch to into pike release if possible
15:59:55 <mkolesni> gtg bye guys
15:59:57 <yamahata_> rajivk: I see. okay.
16:00:14 <rajivk> did you check failure of CI?
16:00:30 <yamahata_> Yes, now I'm addressing CI breakage for now.
16:00:41 <yamahata_> and manjeets is looking at floating ip issue.
16:00:52 <yamahata_> We're also talking is netvirt people.
16:01:07 <yamahata_> Off course it's very glad for you to jump in.
16:01:22 <yamahata_> we run out of time.
16:01:27 <rajivk> i went through discussion, it seems now floating ip issue is resolved. as per last comment from manjeet ODL bug.
16:01:28 <yamahata_> thank you everyone
16:01:48 <yamahata_> rajivk: yeah. after fixing CI breakage, we'll see the result.
16:02:05 <rajivk> i tried on the same day to verify it
16:02:24 <rajivk> but i was not able to verify it.
16:02:33 <yamahata_> Ohh,
16:02:37 <rajivk> I checked you last patch for bug fix
16:03:10 <yamahata_> rajivk: please put your comment on the review. then we can continue discussion and investigate the issues.
16:03:22 <rajivk> yamahata_, ok
16:03:33 <yamahata_> manjeets: are you still there?
16:03:36 <manjeets> yes
16:03:43 <yamahata_> please follow it up.
16:03:58 <yamahata_> anything else?
16:04:11 <rajivk> no
16:04:19 <yamahata_> okay thank you!
16:04:25 <yamahata_> #topic cookies
16:04:27 <manjeets> yamahata_, what the status of ci after neutron sg patch merge ?
16:04:33 <yamahata_> #endmeeting