15:02:36 <yamahata> #startmeeting neutron_northbound
15:02:36 <odl_meetbot> Meeting started Wed Apr 20 15:02:36 2016 UTC.  The chair is yamahata. Information about MeetBot at http://ci.openstack.org/meetbot.html.
15:02:36 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:36 <odl_meetbot> The meeting name has been set to 'neutron_northbound'
15:02:42 <yamahata> #topic agenda bashing and roll call
15:02:47 <yamahata> #info yamahata
15:03:00 <yamahata> #link https://wiki.opendaylight.org/view/NeutronNorthbound:Meetings agenda page
15:03:15 <yamahata> today we don't have update on meeting page.
15:03:25 <yamahata> Do we have any topics for today?
15:03:33 <yamahata> It's less than 1 week for openstack summit
15:03:53 <yamahata> #chair asomya mkolesni oshvartz
15:03:53 <odl_meetbot> Current chairs: asomya mkolesni oshvartz yamahata
15:04:02 <mkolesni> just reviews on my side
15:04:15 <oshvartz> oshvartz, Same for me
15:04:49 <yamahata> okay, move on
15:04:55 <yamahata> #topic Announcements
15:05:15 <yamahata> next week openstack summit will be held.
15:05:27 <yamahata> So this meeting will be skipped next week.
15:05:43 <yamahata> For neutron developer
15:05:49 <yamahata> #link https://www.openstack.org/summit/austin-2016/summit-schedule/global-search?t=Neutron%3A
15:06:00 <yamahata> #link https://wiki.openstack.org/wiki/Design_Summit/Newton/Etherpads#Neutron
15:06:47 <yamahata> And I'm proposing to gather on Monday and Tuesday around the event registration.
15:06:58 <yamahata> during keynot speach
15:07:54 <yamahata> that's all from me. any announcement?
15:08:43 <yamahata> seems nothing else
15:08:51 <yamahata> #topic action items from last meeting
15:08:59 <yamahata> mostly for yamahata
15:09:29 <yamahata> I've proposed odl gathering offline.
15:09:53 <gongysh> yamahata, surprised to know the meeting is on this channel.
15:09:58 <yamahata> I haven't contacted Kyle for netorking-odl release because there is ci issue now.
15:10:03 <yamahata> gongysh: hello.
15:10:14 <yamahata> It's for historical reason.
15:10:34 <yamahata> gongysh: can you please update meeting calendar for openstack?
15:11:06 <gongysh> yamahata, do you mean my own calendar or somewhere?
15:11:10 <yamahata> Once those issues are fixed, I'll contact Kyle. Or meet him at  openstack summit
15:11:30 <yamahata> gongysh:  I mean https://wiki.openstack.org/wiki/Meetings
15:11:41 <yamahata> we need to send a patch.
15:11:56 <yamahata> http://eavesdrop.openstack.org/
15:12:27 <yamahata> gongysh: can we discuss this issue at open mike?
15:12:36 <gongysh> yamahata, sure
15:12:47 <yamahata> okay let's move on bugs/patch
15:12:51 <yamahata> #topic patches/bugs
15:13:09 <yamahata> For now openstack ci doesn't run properly.
15:13:20 <mkolesni> also intel ci seems to fail a lot
15:13:43 <yamahata> With https://review.openstack.org/#/c/307933/ "move beryllium snapshot to 4.2 from 4.1" beryllium  snapshot and boron snapshot runs
15:14:08 <yamahata> But lithium-snapshot CI doesn't run tempest properly.
15:14:45 <yamahata> Please notice its runtime is 20-30min. It setup devstack but fails to run tempest.
15:15:31 <yamahata> I'd like to raise ODL patch issue.
15:15:36 <yamahata> #link https://trello.com/c/r2qbjSHj/86-enable-maven-site-generation
15:15:59 <yamahata> as ODL TSC decided, ODL neutron northbound should follow it.
15:16:29 <yamahata> That's it from me.
15:16:34 <yamahata> Any other patches/bugs to discuss?
15:16:39 <mkolesni> yes
15:16:49 <mkolesni> #link https://review.openstack.org/#/c/305132/
15:17:14 <mkolesni> Add journal maintenance thread
15:17:26 <mkolesni> i don't have much to add there for now
15:17:49 <mkolesni> i think we can proceed with it as is to have the basic maintenance thread functionality there
15:17:56 <mkolesni> so i'll be happy for some reviews
15:18:18 <oshvartz> I tested it and also added a patch with cleanup operation
15:18:49 <yamahata> Okay. How do we want to coordinate with https://review.openstack.org/#/c/304168/
15:19:03 <yamahata> #link https://review.openstack.org/#/c/304168/ Journal entries can get stuck forever causing busy wait
15:19:08 <gongysh> yamahata, do we have any BP spec for journal feature?
15:19:31 <gongysh> it seems this project does not follow normal development process.
15:19:33 <yamahata> gongysh: no BP. but we have a page. https://wiki.opendaylight.org/view/NeutronNorthbound:NeutronDriverOverhaul
15:20:05 <yamahata> I think we can merge 304168 (after review) and then we can convert it into maintenance thread.
15:20:13 <mkolesni> as I wrote initially I think that 304168 should rebase on 305132 and use the maintenance thread infra
15:20:20 <gongysh> yamahata, then we at least, put this URL on review patch.
15:20:24 <mkolesni> then 304168 would be much simpler
15:20:41 <yamahata> asomya: what do you think?
15:21:02 <rcurran> yamahata, asomya had to step away for a few minutes
15:21:12 <oshvartz> I can say that it was very easy to add the cleanup operation from "COMPLETED" rows
15:21:14 <rcurran> perhaps you can come back at end of meeting
15:21:51 <yamahata> gongysh: hmm then it would make sense to create blueprint in launchpad. and add links.
15:22:42 <yamahata> gongysh: Right now networking-odl project doesn't allow to create blueprint. We need to contact Kyle.
15:23:31 <yamahata> rcurran: oh.
15:24:45 <yamahata> Now asomya  is away. can we continue the discussion at 304168 ?
15:25:19 <rcurran> yamahata, i'm suggesting coming back to that question (which patch should go in first) when asomya returns
15:25:32 <yamahata> rcurran: makes sense.
15:25:43 <yamahata> do we have any other patches to discuss?
15:26:16 <oshvartz> #link https://review.openstack.org/#/c/305321/
15:26:34 <oshvartz> Fix delay in sync pending rows
15:27:00 <yamahata> doesn't wait(timeout) work?
15:27:13 <yamahata> I'm a bit surprised.
15:27:24 <oshvartz> No there is an issue with the API workers fork
15:28:13 <yamahata> Ah. So initialization code related API workers needs to be improved, right?
15:28:35 <yamahata> Probably we need something like init before/after API worker fork
15:28:42 <yamahata> don't we?
15:28:51 <mkolesni> theres no support for it in ML2
15:28:57 <mkolesni> from what we saw
15:29:05 <mkolesni> ML2 calls initialize on the driver
15:29:11 <yamahata> Okay, that's good topic to discuss at openstack summit ML2 gathering.
15:29:14 <oshvartz> Yes, but I'm not familiar with that code
15:29:17 <mkolesni> this is before the process fork
15:29:42 <mkolesni> we could add an "initialize_child_process"
15:29:47 <mkolesni> on the whole chain
15:29:51 <rcurran> mkolesni, correct - initialize() method is only called once per driver
15:29:53 <yamahata> can you please put it https://etherpad.openstack.org/p/newton-neutron-unplugged-track
15:30:10 <mkolesni> sure but i'm not at the summit so can't discuss it there
15:30:31 <rcurran> i thought at one point someone was looking into some post neutron-server folks ... long time ago
15:30:34 <yamahata> I'll raise it to ML2 folks, Rob and Shukdev
15:30:57 <rcurran> forks
15:31:09 <oshvartz> I think that for now we can stay with the threading.Timer and once we will have a fix for it we can move to wait(timeout)
15:31:10 <mkolesni> added it to the etherpad
15:31:30 <yamahata> That makes sense.
15:31:31 <mkolesni> rcurran: there were some general neutron issues with that
15:31:42 <yamahata> Can you please add such comment in the commit log?
15:31:44 <mkolesni> but they were fixed, nothing for the drivers tho
15:31:57 <rcurran> mkolesni, oh yeah, lots and lots once people started using api/rpc_workers
15:32:18 <mkolesni> mkolesni: yeah i was the one investigating it on RH side :)
15:32:23 <mkolesni> rcurran: ^^
15:32:31 * mkolesni should stop talking to self..
15:32:50 <oshvartz> :)
15:33:21 <rcurran> a few md drivers had issues w/ multi-neutron processes
15:33:29 <oshvartz> OK, will add a comment on the commit log
15:33:59 <oshvartz> #link https://review.openstack.org/#/c/306818/
15:34:05 <oshvartz> Fix race between event write and thread processing
15:35:24 <mkolesni> this one is rather simple
15:35:27 <yamahata> For me, there is not moot.
15:35:40 <yamahata> (except coding style etc)
15:36:10 <yamahata> I'd like to wait for somya and rcurran to review.
15:36:10 <oshvartz> I fixed the comment on the order of the methods
15:36:38 <rcurran> sorry for delays in reviews ... other work items on plate
15:36:54 <mkolesni> #link https://review.openstack.org/#/c/307218 ODL v2: Full sync resources
15:37:14 <mkolesni> i think this one is quite ready for now
15:37:25 <mkolesni> so i'll be glad to get some reviews
15:37:42 <mkolesni> as you can see it's following the design we discussed on the ML
15:37:44 <oshvartz> I added then to this patch
15:38:10 <yamahata> Cool. will review it.
15:38:39 <mkolesni> i didn't put 2 canary networks yet since i didn't get a feeling we had some sense of what to do there
15:39:04 <mkolesni> i.e. on the ML i think this point was kind of left in the air and there wasn't a decision
15:39:20 <mkolesni> but this can be added quite easily
15:39:39 <mkolesni> also i didnt have time to add the support for "create or update" functionality
15:39:45 <yamahata> okay. it's very details. I think it can be fixed later.
15:39:52 <mkolesni> but i think its best to get reviews first
15:39:53 <yamahata> As long as state machine is clean.
15:39:59 <mkolesni> exactly
15:40:13 <mkolesni> we can always add corner case handling later
15:40:29 <rcurran> mkolesni, this requires https://review.openstack.org/#/c/305132/ to be merged first, correct
15:41:07 <oshvartz> last patch from my side #link https://review.openstack.org/#/c/307171/
15:41:14 <oshvartz> Add cleanup operation to maintenance thread
15:41:14 <mkolesni> rcurran: yes i split it to allow other usage of the maintenance mechanism
15:41:37 <mkolesni> rcurran: also makes it easier to review each independently..
15:41:38 <oshvartz> Also based on 305132
15:42:10 <rcurran> mkolesni, got it and agreed, but need you and asomya to determine which patch goes in first
15:42:28 <rcurran> fyi, he's still not back
15:42:56 <oshvartz> this is a maintenance operation for deleting "COMPLETED" rows
15:43:58 <yamahata> regarding to 307171, I though asomy and rcurran intentionally left COMPLETED rows.
15:44:09 <rcurran> oshvartz, does it delete 'create|update' rows even if a delete (with same UUID) isn't in journal?
15:44:10 <yamahata> rcurran: asomya ?
15:44:28 <mkolesni> yamahata:  it is a problem since the journal will grow indefinitely
15:44:39 <mkolesni> making db access slower
15:44:41 <yamahata> instead we can introduce cleanup script which can be run manually.
15:44:49 <mkolesni> also mass operations can flood the table
15:45:01 <rcurran> mkolesni, yes understood, we knew this was needed.
15:45:03 <yamahata> mkolesni: yes. Do you observing slowdown?
15:45:37 <mkolesni> i don't see why you'd want that but you can have 0 mean the cleanup is off
15:45:44 <mkolesni> or -1
15:45:54 <mkolesni> 0 would probably mean cleanup is immediate
15:46:17 <mkolesni> yamahata: imagine on a large cloud with 10000's or resources this table will get filled fast
15:46:28 <mkolesni> yamahata: each port status change will trigger a row
15:46:54 <mkolesni> yamahata: since the main purpose is for active operation this shouldn't be a DWH table
15:47:08 <mkolesni> if cloud admin wants he can set up ETL for that
15:47:53 <yamahata> I'm not against the patch. Instead I'd like to understand the original intention.
15:49:00 <yamahata> we may want to keep some history of rows for debugging.
15:49:18 <mkolesni> ok then as i suggest you can play with this value
15:49:37 <mkolesni> and even have it that negative value means journal is never cleaned
15:49:40 <oshvartz> the value was added to odl config
15:49:42 <yamahata> Probably some tool to dump rows can be introduced later.
15:50:13 <mkolesni> i theory a malicious tenant can bring down a cloud like this
15:50:30 <yamahata> Please note that I'm not objecting the patch itself.
15:50:32 <mkolesni> by simly issuing an endless stream of updates on a port
15:50:45 <mkolesni> ok then we need to think of stability first
15:50:59 <mkolesni> i think logs should be the place for debugging
15:51:00 <yamahata> I'm wondering what is the good way to handle complet entries.
15:51:39 <mkolesni> at the worst you can change the config value to whatever suits you
15:51:46 <yamahata> mkolesni: logs makes sense.
15:51:48 <mkolesni> including turn off the cleanup if you want
15:52:06 <rcurran> mkolesni, yamahata - i think we've always known that we need to eventually remove rows that are no longer needed. but needed to baby step to get initial code working
15:52:34 <mkolesni> rcurran: in that view this patch makes sense then
15:52:38 <rcurran> my first thought would be to only remove those journal rows that have a completed deltete
15:52:49 <rcurran> delete
15:53:09 <mkolesni> what do you mean "a completed delete"?
15:53:28 <oshvartz> That the object was deleted ?
15:53:31 <rcurran> when a delete+object+uuid is complete then remove all rows w/ that UUID
15:54:00 <rcurran> since then we know the controller has deleted the resource
15:54:44 <oshvartz> You don't think that this logic is unnecessary ?
15:54:50 <mkolesni> again youll have very quickly a full table
15:54:54 <mkolesni> with slow access
15:55:13 <mkolesni> lets KISS
15:55:25 <mkolesni> and delete the rows that are completed
15:55:52 <mkolesni> if you're a brave/stupid cloud operator that wants no delete then turn it off
15:56:02 <rcurran> sorry, i haven't looked at the patch. do you remove journal rows that are create|updates (but no deletes)?
15:56:27 <rcurran> based on the entries being there for a (long) time period?
15:56:31 <mkolesni> its removing all rows that are in state = completed
15:56:44 <mkolesni> that have been there more than the configure time
15:56:58 <rcurran> mkolesni, but w/ no 'delete complete' rows for that UUID
15:56:59 <mkolesni> the time is configured in the ini file
15:57:45 <mkolesni> if you mean 'delete complete' that the resource is deleted from the system, then no this is not checked
15:58:04 <mkolesni> and i dont believe theres a reason to do this
15:58:25 <oshvartz> I agree with mkolesni that for debugging we have logs
15:58:39 <mkolesni> for a running system this table should be either very small size, or empty
15:58:51 <mkolesni> otherwise we will hit performance and/or space issues
15:59:04 <rcurran> mkolesni, ok, that's my point. i'm thinking about replay the journal
15:59:18 <yamahata> rcurran: interesting.
15:59:21 <mkolesni> rcurran: for what purpose?
15:59:34 <yamahata> It can be used for testing/debug
15:59:44 <mkolesni> rcurran: for debug just set the time to 999999999
15:59:58 <mkolesni> or -1 to turn it off, if we decide this makes sense
16:00:09 <mkolesni> sorry i have to go
16:00:24 <yamahata> okay, let's continue as review.
16:00:26 <mkolesni> im guessing we will talk in 2 weeks
16:00:34 <mkolesni> enjoy the summit!
16:00:37 <yamahata> lastly I'd like to raise https://review.openstack.org/#/c/295368/
16:00:42 <oshvartz> I think in production you won't want the table to be very bug
16:00:44 <oshvartz> bug
16:00:45 <yamahata> I'd like to include the patch for mitaka release-odl.
16:00:46 <oshvartz> big
16:01:13 <yamahata> #link https://review.openstack.org/#/c/295368/
16:01:17 <yamahata> any other issue?
16:01:23 <yamahata> #topic open mike
16:02:32 <yamahata> okay thank you every one
16:02:38 <oshvartz> I saw that we don't have a tag for a long time
16:02:58 <yamahata> oshvartz: what do you mean by a tag?
16:03:25 <oshvartz> I mean for liberty i didn't see a tag in the git repo
16:03:42 <yamahata> Got it. git tag.
16:03:47 <oshvartz> yes
16:04:47 <oshvartz> When are we planning to do one for Mitaka
16:04:52 <yamahata> I see. It's a problem.
16:05:07 <yamahata> Soon. Once we ask Kyle, we can have a tag/branch.
16:05:21 <yamahata> For future, we should have the controle. I'll talk with Kyle.
16:05:23 <oshvartz> I think we need both
16:05:57 <oshvartz> branch for mitaka and also a tag for the release
16:06:50 <yamahata> I agree.
16:07:14 <yamahata> The current governance has a bit problem. There is no active coredev who has maintenance rights.
16:07:30 <oshvartz> OK
16:07:54 <yamahata> Let's resolve it. I'll send a mail with Cc to you and others.
16:08:05 <oshvartz> great
16:08:20 <yamahata> #action yamahata send a mail to Kyle on governance.
16:08:53 <yamahata> any other issue? we run over several minutes.
16:09:33 <yamahata> okay, thank you.
16:09:42 <yamahata> see you at openstack summit
16:09:46 <yamahata> #topic cookies
16:09:52 <yamahata> #endmeeting