15:03:26 <yamahata> #startmeeting neutron_northbound
15:03:26 <odl_meetbot> Meeting started Mon Mar 27 15:03:26 2017 UTC.  The chair is yamahata. Information about MeetBot at http://ci.openstack.org/meetbot.html.
15:03:26 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:03:26 <odl_meetbot> The meeting name has been set to 'neutron_northbound'
15:03:29 <reedip> sridharg : ping
15:03:36 <yamahata> #topic agenda bashing and roll call
15:03:41 <yamahata> #info yamahata
15:03:49 <reedip> #info reedip
15:03:52 <yamahata> #link https://wiki.opendaylight.org/view/NeutronNorthbound:Meetings meeting agend
15:04:16 <yamahata> in addition to usual topics, any other topic to discuss today?
15:04:27 <rajivk> #info rajivk
15:04:44 <reedip> yamahata : IPv6 ICMP :)
15:04:45 <yamahata> From last week, patches to be discussed and we have several new patches/bug reports.
15:04:57 <yamahata> icmpv6, yeah.
15:06:00 <yamahata> #topic Announcements
15:06:05 <yamahata> any announcement?
15:06:22 <yamahata> ODL Carbon is near to release.
15:06:52 <yamahata> ODL neutron northbound is code freeze. Karaf4 needs one more patch so that integration test is run with karaf4
15:07:10 <yamahata> ODL DDF is announced officially
15:07:44 <yamahata> #link http://events.linuxfoundation.org/events/opendaylight-developer-design-forum ODL ddf
15:08:12 <yamahata> For now, only place and date is announce. the details isn't announced and the registration is not open yet.
15:08:20 <yamahata> Santa Clara, CA
15:08:27 <yamahata> May 31- June 2, 2017
15:08:42 <yamahata> From me, that's it. any other announcement?
15:09:32 <yamahata> seems nothing. move on.
15:09:34 <yamahata> #topic action items from last meeting
15:09:54 <yamahata> only patch review and bug scrub.
15:10:08 <yamahata> #topic Pike planning
15:10:16 <yamahata> We have two spec
15:10:24 <yamahata> in review process
15:10:47 <yamahata> #link https://review.openstack.org/#/c/443541/
15:10:47 <yamahata> Propose spec for dependency validations move
15:10:57 <yamahata> This one is discussed last week
15:11:11 <yamahata> #link https://review.openstack.org/#/c/449402/ spec: rpc from odl to networking-odl
15:11:16 <yamahata> This one is new one I've uploaded
15:12:01 <yamahata> regarding to 443541. Only issue is performance/benchmark. Otherwise there is consensus.
15:12:13 <yamahata> Probably we can merge it after minor review resping.
15:12:22 <yamahata> Regarding to 449402, this is new one I've uploaded.
15:13:15 <yamahata> Probably we need to discuss with actual user. i.e. netvirt dhcp people. Unfortunately we don't have them today.
15:13:33 <mkolesni> hi sorry im late
15:13:39 <yamahata> mkolesni: hello
15:13:47 <yamahata> Now we're discussing pike planning.
15:14:00 <yamahata> we have two spec uploaded under review.
15:14:23 <yamahata> #link https://blueprints.launchpad.net/networking-odl blueprint of networking-odl
15:14:58 <mkolesni> brb
15:14:58 <yamahata> #topic carbon/nitrogen planning
15:15:31 <yamahata> for nitrogen, we need to create release plan soon.
15:16:00 <yamahata> Nothing urgent
15:16:13 <yamahata> #topic bugs/patches
15:16:20 <yamahata> Okay. Now discussion time.
15:16:31 <yamahata> There are several bug reports and tempest failures.
15:16:52 <yamahata> Especially tempest failures are blocking patch backports.
15:17:14 <yamahata> Is anyone looking into it?
15:17:36 <rajivk> yamahata, do you mean Alon patches
15:17:44 <reedip> yamahata: is there a grafanna dashboard ?
15:17:51 <yamahata> rajivk: yes for backporting.
15:18:07 <mkolesni> i havent had a chance to look yet
15:18:13 <yamahata> #link http://grafana.openstack.org/dashboard/db/networking-odl-failure-rate grafana for netowkring-odl
15:18:25 <mkolesni> is everything on stable broken?
15:18:26 <rajivk> yeah, i checked them.
15:18:28 <mkolesni> master seems fine
15:18:38 <yamahata> Yes, even it's broken for master branch.
15:18:41 <reedip> yeah got it yamahata
15:18:48 <rajivk> I will investigate it.
15:18:49 <reedip> wow, py27 is broken
15:18:52 <yamahata> As temporal work around, several test cases are disabled
15:18:57 <reedip> rajivk +1
15:19:21 <rajivk> yamahata, i want clarity on it.
15:19:47 <yamahata> rajivk: sure, what needs clarity?
15:19:54 <rajivk> You said master is broken, i tested today.
15:19:56 <reedip> yamahata: only v1 driver tempest is failing
15:19:59 <rajivk> It installed successfully.
15:20:18 <rajivk> ohh, you mean tempest tests are failing.
15:21:02 <yamahata> Regarding to v1 driver, probably we can ignore/disable it as v1driver is being depricated.
15:21:03 <reedip> rajivk : py27 and tempest both are failing Currently
15:21:09 <mkolesni> thats weird since the change is for v2
15:21:18 <mkolesni> yamahata, im not sure we should
15:21:28 <mkolesni> v2 is default only from ocata
15:21:37 <reedip> yamahata as per grafanna, v1 driver is failing, so we can disable it and then check the tempest status .. what do you think?
15:21:57 <yamahata> mkolesni: makes sense.
15:21:59 <rajivk> mkolesni, +1
15:22:22 <mkolesni> seems like some infra issue there
15:22:25 <reedip> mkolesni : v1 can disabled for pike, isnt it?
15:22:32 <mkolesni> tests didnt even run
15:22:49 <yamahata> Did we file a bug report for it?
15:22:51 <mkolesni> reedip, sure i guess
15:23:00 <rajivk> mkolesni, Yamahata
15:23:06 <rajivk> How many tests are failing?
15:23:16 <mkolesni> rajivk, they didnt even run
15:23:21 <mkolesni> the infra failed
15:23:30 <reedip> rajivk , you can also check http://grafana.openstack.org/dashboard/db/networking-odl-failure-rate
15:23:58 <mkolesni> 2017-03-24 05:12:54.044550 | + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:main:L719:   /tmp/ansible/bin/ansible primary -f 5 -i /home/jenkins/workspace/gate-tempest-dsvm-networking-odl-carbon-snapshot-v1driver/inventory -m shell -a 'cd '\''/opt/stack/new/devstack'\'' && sudo -H -u stack DSTOOLS_VERSION=0.3.0 stdbuf -oL -eL ./stack.sh executable=/bin/bash'
15:23:58 <mkolesni> 2017-03-24 05:12:58.046981 | ERROR: the main setup script run by this job failed - exit code: 2
15:24:22 <reedip> mkolensi: this one ?? http://logs.openstack.org/27/449727/1/check/gate-grenade-dsvm-networking-odl-nv/c10a925/console.html#_2017-03-24_17_24_53_896745
15:24:52 <rajivk> i think, i checked it.
15:25:03 <rajivk> let me verify
15:25:16 <mkolesni> im looking here http://logs.openstack.org/75/448475/5/check/gate-tempest-dsvm-networking-odl-carbon-snapshot-v1driver/59bc7fd/logs/
15:25:38 <reedip> which file mkolesni ?
15:25:55 <mkolesni> look in the console log
15:26:06 <mkolesni> also test_results.html is empty
15:26:11 <mkolesni> its for this change https://review.openstack.org/#/c/448475/
15:26:27 <rajivk> http://logs.openstack.org/75/448475/5/check/gate-tempest-dsvm-networking-odl-carbon-snapshot-v1driver/59bc7fd/logs/devstack-early.txt.gz
15:26:57 <rajivk> mkolesni, it is for newton branch.
15:27:02 <yamahata> the time is only several minutes. So it failed to setup running environment.
15:27:20 <rajivk> Is master also failing with the same error?
15:27:21 <vthapar> hi all, sorry for being late.
15:27:33 <mkolesni> hi vthapar
15:27:42 <yamahata> vthapar: hello
15:27:50 <mkolesni> probably some change in the dependencies of the stable branch caused this
15:29:21 <reedip> rajivk : how did you determine its newton ??
15:29:39 <reedip> any suggestions?
15:30:18 <rajivk> the link mkolesni shared, it is in stable/newton
15:31:08 <reedip> oh , but I looked at the recent logs as well buddy
15:31:27 <rajivk> please share the link
15:32:08 <yamahata> the devstack-early log show that devstack is wating for password for rabbitmq.
15:32:19 <yamahata> the config seems changed.
15:32:32 <rajivk> yamahata, link please
15:32:50 <reedip> yamahata : yep
15:32:50 <yamahata> Oh no. I was wrong?
15:33:01 <yamahata> http://logs.openstack.org/75/448475/5/check/gate-tempest-dsvm-networking-odl-carbon-snapshot-v1driver/59bc7fd/logs/devstack-early.txt.gz
15:33:04 <yamahata> your link.
15:33:08 <reedip> No I found that as well in mkolensi's log
15:34:10 <yamahata> Do we want to continue debug session? or file a bug and move on to other bugs/patches?
15:34:23 <yamahata> anyway I'll volunteer to file a bug
15:34:34 <yamahata> #action yamahata file a bug for stable branch breakage.
15:34:52 <reedip> yes , kets do it after the meeting
15:35:06 <rajivk> yamahata, the discussion we were having, was it for newton or master?
15:36:08 <yamahata> I've thought for both.
15:36:14 <rajivk> master seems to be fine, i submitted a doc patch
15:36:19 <rajivk> It was fine.
15:36:32 <rajivk> does doc patch does not have functional and fulltest?
15:36:34 <rajivk> https://review.openstack.org/#/c/446390/
15:36:41 <yamahata> Ah devstack is okay. there is ssh failure for master
15:36:56 <yamahata> some test cases are disabled for master.
15:37:09 <reedip> rajivk functional is working
15:37:13 <yamahata> devstack fails for stable(ocata, newton) branch
15:37:17 <reedip> http://logs.openstack.org/90/446390/5/check/gate-dsvm-networking-odl-functional-carbon-snapshot/a8618d2/
15:37:44 <reedip> but I think fullstack is not there
15:37:48 <rajivk> So it means, master as of now correct, because some of the test cases are disabled.
15:38:13 <rajivk> yamahata, If we enabled them then master will also fail, is my understanding correct?
15:38:21 <yamahata> rajivk: right.
15:38:43 <rajivk> yamahata, you are talking about https://review.openstack.org/#/c/445251/
15:38:48 <yamahata> https://review.openstack.org/#/c/445251/ doesn't pass
15:38:54 <yamahata> rajivk: yes.
15:39:09 <rajivk> I checked it and put a comment.
15:39:26 <rajivk> May be you can provide your opinion on it.
15:39:49 <rajivk> Did you check my comment. Let's leave this discussion for now.
15:40:20 <yamahata> I haven't checked it deeply. I'll check it.
15:40:41 <yamahata> #action yamahata check rajivk comment on https://review.openstack.org/#/c/445251/
15:41:25 <yamahata> Now can we move on to other bugs/patches?
15:41:37 <rajivk> yes, we can :)
15:41:49 <mkolesni> can i get reviews on remaining patches on https://review.openstack.org/#/q/status:open+project:openstack/networking-odl+branch:master+topic:journal-singleton please?
15:41:55 <mkolesni> #link https://review.openstack.org/#/q/status:open+project:openstack/networking-odl+branch:master+topic:journal-singleton
15:42:19 <mkolesni> i fixed it according to the comments
15:42:28 <yamahata> Cool.
15:42:50 <yamahata> anything else? icmpv6?
15:43:05 <rajivk> wait
15:43:17 <yamahata> reedip?
15:43:22 <reedip> hi
15:44:03 <yamahata> #link https://review.openstack.org/#/c/449825/
15:44:03 <yamahata> secgroup: convert icmpv6 variant name into icmpv6
15:44:20 <yamahata> This patch is to response to neutron discussion at mailing list.
15:44:40 <reedip> icmp and ipv6 can be divided into 3 combinations
15:44:44 <reedip> icmpv6
15:44:51 <reedip> ipv6-icmp
15:44:54 <reedip> and icmp ONLY
15:45:04 <yamahata> It would take a while to fix neutron upstream. But I don't want to leave networking-odl broken until neutron fix.
15:45:10 <reedip> currently neutron supports all 3 options
15:45:48 <yamahata> When was ipv6-icmp introduced?
15:45:56 <yamahata> From Pike or older?
15:46:00 <reedip> but it results into duplicate results
15:46:04 <reedip> Mitaka I guess
15:46:25 <yamahata> Oh, then we may need fix for newton and ocata.
15:46:43 <reedip> yamahata, wait, let me make it sure
15:48:06 <reedip> meanwhile I just want tell you guys
15:48:19 <reedip> that due to multiple values of Ipv6 in SG
15:48:25 <reedip> duplicate rules are present
15:48:34 <reedip> and that is an inconvineance
15:48:44 <reedip> to fix that , I proposed a patch in Neutron
15:48:53 <reedip> OVS support ipv6-icmp
15:48:59 <reedip> but ODL supports ICMPv6
15:49:12 <reedip> so we need to make sure the behavior remains the same
15:49:32 <yamahata> https://review.openstack.org/#/c/259037/ Add popular IP protocols for security group
15:49:50 <yamahata> the patch was merged Feb 5, 2016.
15:50:07 <yamahata> newton and ocata branch needs fix.
15:50:30 <reedip> for Brian from Neutron ( haleyb) mentioned that Neutron would by default add ipv6-icmp
15:50:42 <reedip> and as per sridharg, we need to add icmpv6 for ODL
15:51:22 <yamahata> ODL neutron northbound knows icmpv6, but doesn't ipv6-icmpv6.
15:54:00 <reedip> yamahata : ok , thats what sridarg mentioned
15:54:14 <rajivk> yamahata, mkolesni, I would like to discussion singleton patch from mkolesni.
15:54:39 <yamahata> rajivk: sure. please go ahead.
15:54:55 <rajivk> mkolesni, You mean due to GIL only one thread should be run?
15:54:56 <yamahata> I left a message on https://review.openstack.org/#/c/449825/
15:55:22 <mkolesni> only one thread is run at any given moment
15:55:33 <mkolesni> so no need to have more than one doing journaling
15:55:44 <rajivk> hmm, i agree.
15:56:07 <rajivk> yamahata, now i would like to discuss about threadpool
15:56:12 <mkolesni> lets say if odl is stuck responding then it doesnt help that we have 10 threads
15:56:25 <mkolesni> since they will all be stuck
15:56:37 <mkolesni> so it doesnt really save us anything
15:56:47 <rajivk> as mentioned by yamahata, do we still need threadpool for thread management?
15:56:49 <yamahata> In some cases, ODL MD-SAL takes a long time.
15:56:57 <yamahata> In that case, multiple thread helps.
15:57:05 <mkolesni> and i/o is the only time afaik the gil is released other than plain context switch
15:57:22 <yamahata> That's right.
15:57:30 <mkolesni> if we hit it we can always add thread pool
15:57:33 <yamahata> and http reuqest is IO. db query is io.
15:57:43 <mkolesni> but anyway in real environment youll have multiple api workers
15:57:55 <mkolesni> so each process should have 1 thread no more
15:58:08 <mkolesni> since you can easily get 10's of simultaneous processes
15:58:11 <yamahata> we can have multiple thread.
15:58:25 <yamahata> Why do we have to stick to 1 thread per 1 process?
15:58:43 <yamahata> 10thread per 1 process would be much cheaper.
15:58:44 <mkolesni> again because of gil there is no real multi threading
15:59:07 <yamahata> gil isn't related.
15:59:16 <mkolesni> of course it is
15:59:19 <yamahata> eventlet is greentread and pymysql is full python implementation.
15:59:30 <mkolesni> i say we need currently the sim-plest solution
15:59:34 <yamahata> the eventlet thread is switched at io.
15:59:52 <mkolesni> and later well see if it has bottlenecks we should solve them
15:59:58 <yamahata> solution for what issue?
16:00:07 <mkolesni> for code design
16:00:23 <mkolesni> as code is more complicated it is harder to maintain
16:01:19 <mkolesni> right now if i look at any log i see 10s of messages 'nothing in journal' popping up all the time
16:01:29 <mkolesni> and multiple of them coming from the same process
16:01:40 <mkolesni> very hard to debug such logs
16:01:49 <mkolesni> and its just noise not really helping
16:02:05 <mkolesni> so if we have singleton everything is much simpler
16:02:11 <yamahata> I agree that log is mess.
16:02:20 <mkolesni> and later on we can add thread pool if we see theres a bottleneck there
16:02:32 <yamahata> That's because right now journal threads aren't maintained properly.
16:02:36 <mkolesni> but right now the number of threads is the number of v2 drivers loaded
16:02:53 <mkolesni> right so were basically agreeing here
16:02:53 <yamahata> In that case, thread is much easier to maintain than multiple processes.
16:03:03 <reedip> yamahata : I saw your comment in sridarg's patch on https://review.openstack.org/#/c/449825/
16:03:08 <mkolesni> but multiple processes happens anyway
16:03:14 <yamahata> So that we can implement smart wakeup mechanism.
16:03:22 <yamahata> instead of waking up all.
16:03:27 <mkolesni> you cant avoid it
16:04:01 <yamahata> For example, we can introduce a config for the number of threads.
16:04:12 <mkolesni> ok this i think should be honored by a spec that we can review the design and comment on it
16:04:57 <yamahata> Hmm, it's 4 min over.
16:05:13 <yamahata> we can continue discussion on the patch or spec.
16:05:20 <yamahata> do we have any other urgent bugs/patches?
16:05:51 <rajivk> not urgent but long
16:05:57 <yamahata> rajivk: which one?
16:05:59 <mkolesni> i gtg
16:06:04 <rajivk> standing patch, eliminate network topology
16:06:08 <mkolesni> if nothing else needed from me
16:06:19 <mkolesni> bye guys
16:06:25 <yamahata> mkolesni: thanks
16:06:35 <rajivk> mkolesni, thanks have a nice day.
16:06:45 <mkolesni> thanks you too :)
16:06:54 <yamahata> rajivk: the patch itself is getting in good shape.
16:07:00 <reedip> yamahata : I agree with your comment on  https://review.openstack.org/#/c/449825/
16:07:29 <yamahata> so the question is when to merge. Pike or Queens
16:08:04 <rajivk> do you mean network-topogy one?
16:08:17 <yamahata> rajivk: yes.
16:08:38 <yamahata> https://review.openstack.org/#/c/334250/
16:09:16 <rajivk> yamahata, i want to know more about thread leader election for houskeeping tasks.
16:09:43 <rajivk> what are the tasks that we would taking into consideration.
16:10:11 <yamahata> hostconfig update.
16:10:16 <yamahata> and port status update
16:10:36 <rajivk> currently, i am thinking of using leader election by assigning numbers to each process at startup and later on using rpc to broadcast its heartbeat
16:10:43 <yamahata> tasks that receive info from ODL and update neutron db
16:11:02 <yamahata> we don't want to use rpc between neutron servers.
16:11:09 <rajivk> and then the process having least number will be elected as leader, it is sort of bully algorithm.
16:11:15 <yamahata> We'd like to use db for polling.
16:11:31 <yamahata> something like maintenance thread.
16:11:50 <rajivk> but what do we poll in database?
16:11:58 <yamahata> Yes. polling
16:12:05 <rajivk> we are already moving to make networking-odl as agentless.
16:12:21 <yamahata> the leader election can be slow. we don't need fast take over.
16:12:26 <yamahata> and just poll db.
16:12:37 <rajivk> ok, what do we poll from db?
16:12:52 <rajivk> on what information, we elect leader?
16:13:13 <yamahata> OpendaylightMaintenance
16:13:21 <yamahata> or new db table.
16:13:44 <rajivk> You mean, we create a new db table for maintaince thread leader election process?
16:14:03 <yamahata> yes. Probably we can OpendaylightMaintenance
16:14:12 <yamahata> we can use OpendaylightMaintenance
16:15:18 <rajivk> you mean https://github.com/openstack/networking-odl/blob/master/networking_odl/db/models.py#L45 this one?
16:15:32 <yamahata> yes
16:15:39 <yamahata> Now it's dedicated for maintenance thread.
16:15:56 <yamahata> Probably we can add one column like task or something
16:16:19 <rajivk> okay, i will check code from this perspective.
16:16:49 <rajivk> BTW i would like to know, why don't you want to use rpc?
16:17:05 <yamahata> So far we haven't used rpc.
16:17:21 <yamahata> we don't want to rely on extra mechanism.
16:17:35 <yamahata> and we have already OpendaylightMaintenance
16:17:46 <rajivk> But all the services rely on messsaging service
16:18:14 <rajivk> they communicate through messaging then why not?
16:18:35 <yamahata> Technically we can.
16:19:00 <rajivk> But there must be some factors because of which we prefer database but not messaging?
16:19:06 <rajivk> what are those factors?
16:20:43 <yamahata> Anyway after rpc, we need to keep track of their liveness.
16:20:50 <yamahata> something similar of agentdb
16:21:13 <yamahata> It's OpendaylightMaintenance
16:21:18 <rajivk> actually we need to broadcast messages only for aliveness.
16:21:25 <yamahata> without rpc, each neutron server can directly update the table.
16:21:49 <yamahata> Yes.
16:22:38 <yamahata> we don't need strict leader election. only easy implemetation is wanted.
16:22:53 <rajivk> okay
16:23:31 <yamahata> so I've thought we can just slightly enhance the existing maintenance thread one.
16:23:34 <rajivk> I will consider the table you mentioned.
16:24:31 <rajivk> This meeting is not ended yet :)
16:24:39 <yamahata> oh let's end it
16:24:45 <yamahata> #topic open mike
16:24:46 <rajivk> ok
16:24:51 <yamahata> #topic cookies
16:24:55 <yamahata> #endmeeting