13:29:53 <dmcbride> #startmeeting OPNFV release daily
13:29:53 <collabot`> Meeting started Mon Mar 27 13:29:53 2017 UTC.  The chair is dmcbride. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:29:53 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
13:29:53 <collabot`> The meeting name has been set to 'opnfv_release_daily'
13:30:05 <dmcbride> #topic roll call
13:30:12 <dmcbride> #info David McBride
13:30:27 <yujunz> #info Yujun Zhang
13:30:34 <chigang> #info Justin chi
13:30:34 <dmcbride> narinder: are you available?
13:31:10 <narinder> dmcbride, yes
13:31:21 <narinder> #info Narinder Gupta
13:31:47 <dmcbride> #topic scenario status
13:32:13 <dmcbride> narinder: looks like your k8 scenarios are having difficulty
13:32:25 <dmcbride> narinder: are you still planning to release those with D1?
13:32:53 <dmcbride> chigang: I see problems with two of your scenarios on yardstick:
13:32:58 <dmcbride> os-odl_l2-nofeature-ha
13:32:58 <dmcbride> os-odl_l3-nofeature-ha
13:33:23 <narinder> dmcbride, deployment works some how some other issue https://build.opnfv.org/ci/job/joid-deploy-baremetal-daily-danube/53/
13:33:49 <georgk> #info Georg Kunz
13:34:18 <dmcbride> narinder: when I checked yesterday, k8-nosdn-lb-noha was failing deploy
13:34:25 <dmcbride> narinder: is it passing now?
13:34:46 <narinder> check this one it is passed https://build.opnfv.org/ci/job/joid-deploy-baremetal-daily-danube/53/console
13:34:52 <narinder> i have not checked the latest
13:34:53 <chigang> dmcbride: yes, we are  in troubleshooting,  but not failed in every time.
13:35:09 <narinder> but looks like this is avialaiblity of lab issue
13:35:23 <trozet> i'm here btw..sorry late
13:35:59 <dmcbride> chigang: os-odl_l3-nofeature-ha had failed yardstick every time (0/10) when I looked late yesterday
13:36:13 <dmcbride> trozet: hi Tim, glad you could join
13:37:18 <dmcbride> trozet: or frankbrockners fdio scenarios seem to be declining on functest
13:37:37 <chigang> dmcbride: sorry, I thought you mentioned functest
13:37:41 <dmcbride> trozet: or frankbrockners any idea what's going on there
13:38:06 <dmcbride> chigang: no, I was referring to yardstick
13:38:14 <trozet> dmcbride: the issue is the fdio scenarios seem to fail on lfpod1 for testing, but pass in other environments
13:38:31 <chigang> dmcbride: OK, I will check it tomorrow.
13:38:48 <trozet> dmcbride: and if i login to the lfpod1 and test it manually it works, so we are not quite sure what the problem is yet.  I have requested more debug logs from functest when a failure occurs so I can try to investigate
13:39:14 <dmcbride> chigang: os-odl_l3-nofeature-ha seems to be degrading over time on functest, if you look at the spreadsheet I sent yesterday
13:39:28 <trozet> sorry I should have bene using info
13:39:32 <trozet> let me fix it
13:39:51 <narinder> dmcbride, also k8 required to remove the functest and yardstick which is not valid.
13:39:53 <trozet> #info fdio functest seems to fail in lfpod1, but passes in other environments...investigating the cause
13:39:59 <trozet> #info for apex that is
13:41:17 <dmcbride> frankbrockners: are you aware of this issue that Tim has identified?
13:41:46 <trozet> dmcbride: yeah he is aware, we have some FDS engineers also trying to reproduce it
13:42:43 <dmcbride> #info dmcbride notes problems with compass scenarios os-odl_l2-nofeature-ha and os-odl_l3-nofeature-ha
13:43:05 <dmcbride> #info chigang will investigate
13:43:08 <trozet> #info Apex has os-nosdn-ovs-ha and os-odl_l3-ovs-ha working, adding it to daily for todays run
13:43:33 <chigang> dmcbride: as I mentioned before, there are some testcases failed sometimes, it is not too easy to fix.
13:43:43 <chigang> dmcbride: I will check them
13:43:46 <dmcbride> pma: are you available
13:43:59 <pma_> dmcbride> yep
13:44:05 <dmcbride> chigang: that's why you get paid the big bucks :)
13:44:34 <dmcbride> pma_:  seems like we are seeing a lot of problems with Fuel
13:44:48 <pma_> we've issue w/ yardstick against all odl scenarios https://jira.opnfv.org/browse/YARDSTICK-603
13:45:24 <pma_> yeah, plus upstream bug https://bugs.launchpad.net/fuel/+bug/1676197
13:45:26 <dmcbride> #info dmcbride says that only one candidate scenario on Fuel was passing as of yesterday
13:45:57 <dmcbride> pma_: is that bug a root cause, or just one of a number of problems?
13:46:17 <pma_> dmcbride> it's blocker for all deployments
13:46:32 <dmcbride> #link https://bugs.launchpad.net/fuel/+bug/1676197
13:46:44 <pma_> dmcbride> we're working on it, fix is coming ...
13:46:47 <dmcbride> pma_: is this an issue you've told me about before?
13:47:19 <dmcbride> #info pma_ says that this is a blocker for all deployments ^^
13:47:20 <pma_> dmcbride> no, it's new, related to the feature of distributed workes
13:48:37 <dmcbride> narinder: I checked again and k8-nosdn-lb-noha still does not appear to be passing deploy
13:49:47 <narinder> dmcbride, i am seeing something different r u seeing by color or looking for deployment status. Check this link https://build.opnfv.org/ci/view/joid/job/joid-deploy-baremetal-daily-danube/
13:50:33 <narinder> if it is color then it might be scheduling issues as i was seeing multiple jobs were schedules but no executor
13:50:53 <narinder> so i have to abort few of the scenarios today
13:50:55 <dmcbride> narinder: I'm looking at this:  https://build.opnfv.org/ci/job/joid-k8-nosdn-lb-noha-baremetal-daily-danube/
13:51:31 <narinder> dmcbride, march 24th deployment got passed
13:51:51 <narinder> this is detailed deployment job https://build.opnfv.org/ci/view/joid/job/joid-deploy-baremetal-daily-danube/53/console
13:52:30 <dmcbride> narinder: ok - thanks
13:52:41 <narinder> dmcbride, also in this https://build.opnfv.org/ci/job/joid-k8-nosdn-lb-noha-baremetal-daily-danube/ deployment was passed
13:52:55 <narinder> see the first bullet which is blue color
13:54:07 <dmcbride> narinder: ok - got it
13:54:18 <trozet> #info Apex: os-odl_l3-feature-ha functest was fixed, I see the trend score moving upwards
13:54:21 <narinder> dmcbride, cool thanks
13:54:25 <dmcbride> narinder: I'm still figuring out how to read Jenkins
13:55:04 <narinder> dmcbride, it is confusing for me sometimes. Even though it should be straight forward but not some times.
13:56:07 <dmcbride> trozet: that is consistent with the spreadsheet
13:56:45 <dmcbride> trozet: 15 ==> 21 ==> 26
13:56:52 <trozet> dmcbride: from Apex side, i think our main focus right now is fixing FDIO and bgpvpn scenarios
13:57:54 <jose_lausuch> trozet: Nikolas is looking into bgpvpn I think
13:58:11 <dmcbride> trozet: I also see that yardstick is either not running or is failing on most Apex scenarios
13:58:18 <trozet> jose_lausuch: ok cool, was just looking to see if he was online to ping him :)  I think he filed a bug that quagga wasnt coming up right
13:58:41 <trozet> dmcbride: i havent really looked at yardstick at all yet, been focused on functest
13:59:04 <trozet> radez: ping?
13:59:54 <radez> hey trozet
14:00:14 <dmcbride> #info narinder notes that dmcbride read the deploy data in Jenkins wrong for k8-nosdn-lb-noha
14:00:29 <dmcbride> #action dmcbride to update spreadsheet
14:01:20 <trozet> radez: can you look through apex yardstick results on the daily job and try to determine why they are not passing?
14:01:32 <trozet> radez: maybe start with os-nosdn-nofeature as that one is passing almost 100% functest
14:02:36 <radez> trozet: won't be able to do today, but I can put it on my list for first thing tomorrow
14:02:55 <trozet> radez: ok thanks
14:03:05 <trozet> #action radez to look  into Apex yardstick failures in daily
14:03:26 <trozet> radez: while we got you here, want to give an update on OVN ha?
14:04:52 <dmcbride> trozet: is radez on another channel?
14:05:10 <trozet> dmcbride: he's in here
14:05:16 <dmcbride> trozet: I'm not seeing his input on opnfv-release
14:05:25 <trozet> dmcbride: oh, i see it
14:05:25 <dmcbride> trozet: or is he just lurking ;)
14:05:39 <frankbrockners> trozet: sorry - in a IETF meeting right now - Michal is investigating why L3 failed deployment - we did not really change anything
14:06:22 <dmcbride> frankbrockners: trozet thinks the fdio failures may be pod-specific
14:06:58 <trozet> frankbrockners: we can talk about that one in the meeting, i think we are more concerned about fdio odl_l2 noha failing functest
14:07:07 <trozet> frankbrockners: it passes in cengn lab
14:07:26 <trozet> frankbrockners, dmcbride: i'm going to kill the daily when i see the functest start for that scenario and go try to debug
14:08:03 <frankbrockners> trozet: yes - I asked jlinkes as well... - let's nail this in the FDS meeting
14:08:04 <dmcbride> frankbrockners: any other updates for us before we end the meeting?
14:08:09 <trozet> jose_lausuch: fyi we could use some more log collection when the healthcheck fails.  Did you ever get that added to functest?  Should we be running with DEBUG true?
14:08:43 <frankbrockners> trozet: could you also enable nosdn-fdio-noha in the pipeline... the nosdn-fdio-ha scenario makes progress, but some issues remain...
14:08:44 <radez> sry, was distracted for a min, we will have OVN non-HA for D1
14:08:55 <jose_lausuch> trozet: what kind of logs? from the deployment as such?
14:08:59 <trozet> frankbrockners: yes I will enable it
14:09:04 <radez> I'm not sure about OVN-HA yet still investigating it
14:09:06 <frankbrockners> trozet: thanks
14:09:09 <jose_lausuch> trozet: nova.log and so? or functest logs?
14:09:26 <dmcbride> jose_lausuch: that reminds me.  I will bring this up in the TWG, but I think that it would be useful to display a timestamp for each scenario in the functest dashboard
14:09:33 <radez> trozet: in fact I think I'm going to pull out the senario file for ovn-ha and get the non-ha verified and merged before I continue with the ha stuff
14:09:38 <trozet> jose_lausuch: right, we see dhcp fail, and what we really need are some logs from neutron, and nova, and vpp.  We also need a virsh dumpxml of the instance so we can verify it has a vhostuser port
14:09:56 <trozet> radez: that works, we can just claim support for the noha version and add it to the daily
14:10:04 <jose_lausuch> trozet: I started working on it before my 2 weeks off, I plan to continue this week
14:10:26 <jose_lausuch> dmcbride: you mean duration?
14:10:52 <dmcbride> jose_lausuch: time stamp for last run
14:12:14 <dmcbride> jose_lausuch: if I remember correctly, the data covers a 10 day window, but you have no idea if the most recent run was hours ago or days ago
14:12:15 <jose_lausuch> dmcbride: we have a link to the last run of you click on it
14:12:18 <radez> trozet: and fyi we'll have ovn-ha support no problem in E, the issue with HA appears to be puppet code discrepancies in the backports for tht and puppet-ooo
14:12:26 <jose_lausuch> dmcbride: ok, I get it now, yes  we can add that maybe
14:12:54 <trozet> radez: sure if ovn wants HA then they can help us add it, we've done our part
14:13:08 <trozet> radez: s/we/you/
14:13:20 <radez> ack
14:17:12 <dmcbride> no daily meeting tomorrow, since we will have the regular weekly release meeting
14:17:26 <dmcbride> so, the next daily meeting will be on Wednesday at the same time
14:18:04 <dmcbride> #info next daily release meeting on Wednesday at 6:30 a.m. Pacific
14:18:29 <dmcbride> #endmeeting