15:05:01 <yamahata> #startmeeting neutron_northbound
15:05:01 <odl_meetbot> Meeting started Mon Mar 20 15:05:01 2017 UTC.  The chair is yamahata. Information about MeetBot at http://ci.openstack.org/meetbot.html.
15:05:01 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:05:01 <odl_meetbot> The meeting name has been set to 'neutron_northbound'
15:05:07 <yamahata> #chair vthapar
15:05:07 <odl_meetbot> Current chairs: vthapar yamahata
15:05:14 <yamahata> #topic agenda bashing and roll call
15:05:18 <yamahata> #info yamahata
15:05:21 <vthapar> #info vthapar
15:05:29 <rajivk_> #info rajivk
15:05:34 <mkolesni> #info mkolesni
15:05:52 <yamahata> Is there any additional topic toay?
15:05:52 <mkolesni> hi might be a bit busy during the meeting but ill try to attend fully
15:06:39 <yamahata> seems no special topics.
15:06:47 <yamahata> #topic Announcements
15:07:01 <yamahata> ODL carbon M5 mile stone has been reached.
15:07:16 <yamahata> Now it's code freeze.
15:07:31 <yamahata> Only bug fix is allowed.
15:07:48 <yamahata> Karaf4 migration isn't fully done yet. It will be addressed by Dileep.
15:07:59 <yamahata> any other announcement?
15:08:20 <yamahata> #link https://wiki.opendaylight.org/view/NeutronNorthbound:Meetings meeting agenda
15:09:12 <yamahata> move on
15:09:23 <yamahata> #topic action items from last meeting
15:09:38 <yamahata> yamahata send a mail for planning IRC discussion on https://review.openstack.org/#/c/407784/
15:09:44 <yamahata> vthapar to file RFE bug for pre/post test hooks for ODL in tempest
15:09:59 <yamahata> yamahata sent out a mail on 407784. Today we have mkolesni
15:10:23 <yamahata> vthapar: did you file a rfe bug?
15:10:45 <vthapar> yamahata: yes, jsut fetching the bug ID... laptop bit slow due to ODL running in background.
15:10:58 <yamahata> I see thanks
15:11:27 <vthapar> #done vthapar filed https://bugs.launchpad.net/networking-odl/+bug/1672620
15:12:02 <yamahata> thanks for the link
15:12:09 <yamahata> #topic Pike planning
15:12:39 <yamahata> So far we have several spec proposal.
15:12:56 <mkolesni> brb
15:13:01 <yamahata> #link https://review.openstack.org/#/c/443541/
15:13:41 <yamahata> For tech detail we can discuss there.
15:13:51 <yamahata> #link https://blueprints.launchpad.net/networking-odl
15:14:22 <yamahata> Please update your blueprint status and upload spec if necessary
15:14:54 <yamahata> Although I haven't write spec, I'm going to propose one, rpc from ODL.
15:15:15 <yamahata> https://git.opendaylight.org/gerrit/#/c/53242/
15:15:39 <yamahata> vthapar: you may be interested in it. and we'd like to discuss on it at ODL DDF.
15:15:46 <vthapar> I was PTO most of last week being unwell, will get to adding blueprints/specs if any this week.
15:16:06 <yamahata> vthapar: thanks. take care of yourself first.
15:16:10 <vthapar> yamahata: +1
15:16:35 <vthapar> travel plans not finalized yet but most likely will be at DDF and not in Boston.
15:16:52 <yamahata> any blueprint/spec to discuss specifically?
15:17:08 <mkolesni> back
15:17:19 <rajivk_> yes
15:17:30 <vthapar> yamahataL what are plans with RPC one?
15:17:38 <vthapar> yamahata: plans for RPC one?
15:17:52 <vthapar> as in, use cases.
15:18:05 <yamahata> allow ODL to create dhcp port.
15:18:32 <mkolesni> #link https://review.openstack.org/#/c/443541/
15:18:37 <vthapar> aha, so ODL would be like an agent but no heartbeat?
15:18:58 <mkolesni> please review I commented on it and updated some things due to the last review
15:19:02 <yamahata> vthapar: That's right. Now ODL is becoming like agent more and more.
15:19:49 <yamahata> #action everyone review https://review.openstack.org/#/c/443541/
15:20:06 <rajivk_> mkolesni, hi
15:20:14 <mkolesni> rajivk_, hey, sup
15:20:24 <yamahata> mkolesni: do you have anything specific to discuss /explain today?
15:20:33 <rajivk_> I think, i reviewed it and find it in good shape :)
15:20:45 <mkolesni> rajivk_, thanks :)
15:20:56 <mkolesni> lets talk about 407784 ?
15:21:06 <rajivk_> Do you think, adding an example in spec will help to understand better, however it is very well explained.
15:21:07 <mkolesni> kind of RFE really.. not a bug fix
15:21:28 <mkolesni> rajivk_, what you mean an example?
15:21:36 <yamahata> I'm concerned that although it's claiming to improve cpu usage, but no benchmark.
15:21:56 <rajivk_> I mean take a table and dependencies table you are proposing
15:21:58 <yamahata> i.e. no plan to verify it.
15:22:06 <rajivk_> fill in some entries with random examples
15:22:17 <yamahata> Otherwise it seems good experiment for improvement.
15:22:23 <mkolesni> yamahata, as i said if we have the postmortem tool i can run it before and after
15:22:30 <mkolesni> say on the tempest logsa
15:22:31 <mkolesni> logs
15:22:41 <mkolesni> and see what it gives
15:22:53 <vthapar> mkolesni: I just noticed the spec, just omake sure my quick read is correct, aim is to do check before adding to journal and record dependency information in journal itself so we can query, correct?
15:22:57 <mkolesni> if thats the concern
15:23:30 <mkolesni> vthapar, the tl;dr is to calculate the dependencies when we add the entry based on the state of things at that point
15:23:43 <mkolesni> vthapar, instead of trying to guess it later
15:23:43 <yamahata> No. I'm concerned about cpu usage etc.
15:23:58 <rajivk_> mkolesni, do you think, we can make some kind of dependency graph and keep it in memory.
15:24:00 <mkolesni> yamahata, im not sure why you think cpu usage will increase
15:24:06 <yamahata> We need to check if cpu usage is improved with the change by a sort of benchmark.
15:24:22 <yamahata> mkolesni: you're claiming so by line 43-47
15:24:42 <vthapar> mkolesni, I think yamahata's concern is claiming in spec without means to validate it
15:24:46 <mkolesni> i claim it will probably improve things from a design perspective
15:25:25 <mkolesni> for sure you can't think this will be worse than current situation where dependencies are checked >= 1 times for each entry
15:25:39 <mkolesni> whereas I'm proposing a change that checks = 1 times
15:26:14 <mkolesni> not sure how in this scenario you imagine that more cpu can ever be used when we're performing a set number of operations instead of a variable number
15:26:34 <yamahata> extrawork is added at entry creation time.
15:26:46 <mkolesni> no
15:27:03 <mkolesni> work is moved from entry selection time to entry creation time
15:27:22 <mkolesni> there will be no dependency calculation done on entry selection
15:27:55 <mkolesni> only an additional condition on db query which of course could be optimized if necessary but i dont see a reason to look into it now
15:28:29 <mkolesni> i.e. where not exists (select 1 from dependency_table where dependent_id = row.seqnum)
15:28:32 <yamahata> the number of potential number of dependent row would be larger at creation time than selection time.
15:28:35 <mkolesni> or something of the sort
15:28:50 <yamahata> So we're not certain for it.
15:29:00 <mkolesni> what do you mean?
15:29:01 <yamahata> The golden rule of optimisation is to measure it.
15:29:27 <mkolesni> look the change comes to address an architectural need im sorry if i wasn't clear in the spec
15:29:30 <yamahata> At least we should do some sort of benchmark.
15:29:52 <yamahata> it can be easy one. it doesn't have to be extensive one.
15:29:54 <mkolesni> i don't care about any possible performance improvement or impact though i don't see how there will be one
15:30:28 <mkolesni> im more interested in a way to know on every second what exactly each entry depends on
15:30:51 <yamahata> That's great.
15:30:55 <mkolesni> and doing this at create time is vastly better since at that point you know the state of things youre working with
15:31:00 <yamahata> Why do you not want to do benchmark?
15:31:17 <mkolesni> doing it on entry selection the image already changed from creation time so the data might be stale
15:31:43 <mkolesni> benchmark is just not important its not the main point of this change
15:31:58 <mkolesni> we need to do benchmarks in general for numerous scenarios
15:32:13 <mkolesni> but i dont see why we need one specifically for the spec
15:32:30 <mkolesni> but i can remove any mention of performance impact if it will be better
15:32:35 <yamahata> I agree that we need benchmark in general.
15:32:48 <yamahata> I'm not claiming extensive one. Just simple one will be okay.
15:32:59 <mkolesni> there is no simple benchmark
15:33:13 <yamahata> run rally and see it's cpu usage.
15:33:17 <yamahata> It would be easy one
15:33:24 <mkolesni> benchmarking is always complicated since it's hard to really measure what you want or even define what you want to measure
15:34:22 <mkolesni> i dont see what that gives you how will you know it improved anything
15:34:34 <mkolesni> lets say cpu usage is 10% higher, then what?
15:34:48 <yamahata> The assumption that when selecting entry, the number of potintal dependent row is very small.
15:34:49 <mkolesni> maybe run time of rally decreased?
15:35:27 <yamahata> the possibility of retry again would be low.
15:35:37 <mkolesni> the idea is wrong to do the dependency calculations on select as i said the state of things changed so you might be getting stale data
15:35:44 <rajivk_> mkolesni, i think, i can take this task of benchmarking. and yamahata can guide me.
15:35:44 <yamahata> If it's large, there is something anomaly.
15:35:48 <rajivk_> Is that ok?
15:36:03 <mkolesni> did you see the recent logs from full sync testing by community members?
15:36:29 <mkolesni> rajivk_, i dont have a problem with that but i think doing this specifically here is not the main point
15:37:17 <mkolesni> if you see the logs youll see theres some entries retrying more than 100 times due to dependency checks
15:37:30 <yamahata> except cpu usage, I'm quite fun for code cleanup/improvement.
15:37:38 <yamahata> Let's see your outcome.
15:38:13 <yamahata> mkolesni: I'm saying it's anomaly. we should change the logic to select entry.
15:38:50 <yamahata> You've sticked to select entry in roundrobbin manner. it can be changed.
15:39:05 <mkolesni> ok ill write your idea in the spec though i dont believe it
15:39:29 <mkolesni> can we talk about 407784 ?
15:39:51 <yamahata> We've spent too much time on the single one.
15:40:04 <yamahata> Let's move the other one.
15:40:18 <mkolesni> if were on the topic of performance, that change intends a penalty for any running system for the sake of debugging
15:40:37 <mkolesni> whereas a post mortem tool can provide the same data without penalty to an active system
15:40:47 <mkolesni> thats why i -2 it
15:41:06 <mkolesni> its basically putting some kind of profiling in a system
15:41:24 <mkolesni> which no one would do in a production system
15:41:36 <mkolesni> not on a fixed interval at least
15:42:11 <rajivk_> mkolesni, may be we can provide some kind of configurable option, which can be used by deleveper to turn it on and turn it off in production
15:42:27 <rajivk_> mkolesni, what do you think?
15:43:03 <rajivk_> In production admin will always have option to profile from cmd.
15:43:12 <mkolesni> we could though again why do you need it given that you can achieve the same outcome with a post mortem tool?
15:43:26 <mkolesni> which was discussed on that spec
15:44:06 <yamahata> I don't think so. Surely we can log entry creation/deletion. it's hard to track the number of pending rows.
15:44:10 <yamahata> because of log rotation.
15:44:36 <mkolesni> log rotation is controllable externally so i dont see how its a problem
15:45:19 <rajivk_> usually log rotations are set per process
15:45:33 <rajivk_> if we don't set log rotation for profiling use case
15:45:43 <rajivk_> it can increase logs size too much.
15:45:57 <yamahata> and multiple neutron servers are running so the log entry of creation/deletion will be scattered.
15:46:17 <mkolesni> yamahata, you mean in ha env?
15:46:22 <yamahata> mkolesni: yes.
15:46:39 <mkolesni> you can use logstash or probably simple rsyslog to funnel them to a single place
15:47:08 <rajivk_> It will make these tools necessary be in production system.
15:47:15 <yamahata> In that case, I wouldn't say it's easily realisable.
15:47:32 <rajivk_> However some deplyment may have some other mechnism to log.
15:47:50 <rajivk_> I agree with mkolesni, idea that it should not be in production
15:48:03 <rajivk_> and with Yamahata about log rotation.
15:48:07 <mkolesni> the whole idea was for development no?
15:48:24 <mkolesni> i just think this might be useful for debugging real production systems
15:48:33 <mkolesni> but usually there you wont have access to the db
15:48:37 <yamahata> ha environment is also be tested somehow.
15:48:43 <mkolesni> youll usually have just the logs
15:49:55 <rajivk_> yes, but admin might have access.
15:50:06 <rajivk_> However, i don't know, whether people use this tool or not.
15:50:21 <mkolesni> if i have a bug in lp usually i dont have access to the actual prod env
15:50:51 <mkolesni> also it might be a few days or even a month passes before someone looks at it
15:51:00 <mkolesni> that why analyzing logs is a better idea
15:51:18 <yamahata> we can have both tools.
15:51:42 <mkolesni> why would you have that?
15:51:53 <mkolesni> two toold that do the same thing that we need to maintain?
15:52:04 <mkolesni> tools*
15:52:06 <yamahata> My original intention is to check the number of backlog in journal db at runtime.
15:52:28 <rajivk_> log rotation and collection at one place makes it difficult to analyze logs.
15:52:31 <mkolesni> ok great so run your log analyzer periodically then
15:52:34 <yamahata> Once we're satisfied with log analyzing tools, we can remove the tool that accesses db.
15:52:51 <rajivk_> Can not we think, about some other way of achiving the same results?
15:53:42 <rajivk_> Something better way, where everyone's point get's resolved.
15:53:50 <mkolesni> how would the tool access db? does it not need the db user/pass?
15:54:16 <yamahata> Yes, it needs them.
15:54:20 <rajivk_> May be conf file is available.
15:54:34 <yamahata> there's 5 min left.
15:55:02 <yamahata> can we continue discussion at gerrit and move on to other topics/patches/bugs?
15:55:09 <mkolesni> sure
15:55:14 <rajivk_> +1
15:55:24 <yamahata> #topic carbon/nitrogen planning
15:55:45 <yamahata> no big one. Carbon is approaching to end. Nitrogen planning will be discussed at ODL DDF in May
15:55:49 <yamahata> #topic patches/bugs
15:55:51 <mkolesni> unfortunately its holiday here while the DDF so i wont be able to make it there
15:55:54 <yamahata> any other patches/bugs
15:56:07 <yamahata> mkolesni: Ohhh that's unfortunate.
15:56:19 <yamahata> At least there is tempest failures related to ssh.
15:56:21 <rajivk_> yamahata, sorry for asking a lot. But launchpad does not seems to be updated.
15:56:36 <mkolesni> also since the summit talks werent accepted i probably wont be going
15:56:37 <yamahata> rajivk: which one?
15:56:57 <rajivk_> there are some critical bugs.
15:57:08 <yamahata> rajivk: I see.
15:57:27 <yamahata> #action yamahata check/scrub launchpad bugs
15:57:38 <yamahata> #link https://review.openstack.org/#/c/445251/ ssh failures
15:57:58 <yamahata> rajivk: has checked the log
15:58:21 <rajivk_> yes, actually machine's were not getting ip/s as far as i could understand logs.
15:58:25 <yamahata> It's a critical issue and blocking ocata, newton backports.
15:59:14 <mkolesni> so you know why its happening?
15:59:28 <mkolesni> seemed like a change on ODL caused this
15:59:29 <rajivk_> no, it has to be investigated.
15:59:43 <rajivk_> I have to check the scenarios etc.
15:59:52 <mkolesni> theyre not getting internal or floating ips?
16:00:04 <rajivk_> But due to lack of n-odl knowledge, it might be difficult for me.
16:00:08 <rajivk_> internal IPs.
16:00:16 <mkolesni> sometimes there are failures due to FIP not being set in ODL correctly but internal IPs work fine
16:00:38 <mkolesni> then dhcp needs to be investigated
16:00:45 <rajivk_> The log i checked, did not have internal Ips.
16:00:52 <vthapar> sorry folks, I need to head out. will check the meeting logs later.
16:00:53 <yamahata> Yeah, I've suspected floatingip. internal ip sounds a bit surprising.
16:01:05 <rajivk_> I have shared link of logs
16:01:06 <mkolesni> bye vthapar
16:01:15 <yamahata> vthapar: bye
16:01:23 <rajivk_> you can verify from the logs
16:01:34 <rajivk_> ifconfig command does not show any IPs.
16:01:48 <rajivk_> because of that ther are no routes added.
16:01:54 <rajivk_> Let me paste the link here.
16:02:10 <mkolesni> rajivk_, did they get ip in the boot? it usually shows in the log from the guest cirros
16:02:17 <rajivk_> #link http://logs.openstack.org/51/445251/1/check/gate-tempest-dsvm-networking-odl-boron-snapshot-v2driver/4293447/console.html#_2017-03-18_07_24_23_683290
16:02:29 <rajivk_> i checked only those logs
16:03:22 <rajivk_> Failure of all the test cases seems to be this only.
16:03:28 <mkolesni> 2017-03-18 07:24:23.610149 | 2017-03-18 07:24:23.609 |     udhcpc (v1.20.1) started
16:03:28 <mkolesni> 2017-03-18 07:24:23.611594 | 2017-03-18 07:24:23.611 |     Sending discover...
16:03:29 <mkolesni> 2017-03-18 07:24:23.613118 | 2017-03-18 07:24:23.612 |     Sending discover...
16:03:29 <mkolesni> 2017-03-18 07:24:23.614824 | 2017-03-18 07:24:23.614 |     Sending discover...
16:03:29 <mkolesni> 2017-03-18 07:24:23.616433 | 2017-03-18 07:24:23.616 |     Usage: /sbin/cirros-dhcpc <up|down>
16:03:30 <mkolesni> 2017-03-18 07:24:23.618135 | 2017-03-18 07:24:23.617 |     No lease, failing
16:03:31 <rajivk_> None of the instances got the IPs.
16:03:38 <mkolesni> yes no getting ip from the dhcp
16:03:48 <yamahata> oh.
16:04:35 <mkolesni> should have ip 10.1.0.6 in this case
16:04:56 <mkolesni> so ODL somehow failed to provide it via dhcp
16:05:06 <mkolesni> but u need to look at ODL log perhaps
16:05:18 <mkolesni> ie karaf log
16:05:45 <rajivk_> How does ODL provides IP, does it get from dhcp namespace or some other mechinsm.
16:06:09 <yamahata> with openstack ci, q-dhcp is running.
16:06:21 <yamahata> ODL sets up only OF rules.
16:06:39 <mkolesni> theres also option for odl to act as dhcp server
16:07:04 <rajivk_> i read somewhere, we have odl expert here.
16:07:11 <mkolesni> as u can see it doesnt fail in beryllium
16:07:33 <mkolesni> so its something that got broke in carbon and backported to boron
16:07:51 <mkolesni> sorry gtg
16:08:00 <mkolesni> thanks guys, have a nice day
16:08:08 <rajivk_> bye, mkolesni
16:08:13 <rajivk_> have a good day :)
16:08:21 <mkolesni> bye, thanks :)
16:08:25 <yamahata> okay thanks everyone
16:08:31 <yamahata> #topic open mike
16:08:42 <yamahata> #topic cookies
16:08:47 <yamahata> #endmeeting