13:29:53 <dmcbride> #startmeeting OPNFV release daily 13:29:53 <collabot`> Meeting started Mon Mar 27 13:29:53 2017 UTC. The chair is dmcbride. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:29:53 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic. 13:29:53 <collabot`> The meeting name has been set to 'opnfv_release_daily' 13:30:05 <dmcbride> #topic roll call 13:30:12 <dmcbride> #info David McBride 13:30:27 <yujunz> #info Yujun Zhang 13:30:34 <chigang> #info Justin chi 13:30:34 <dmcbride> narinder: are you available? 13:31:10 <narinder> dmcbride, yes 13:31:21 <narinder> #info Narinder Gupta 13:31:47 <dmcbride> #topic scenario status 13:32:13 <dmcbride> narinder: looks like your k8 scenarios are having difficulty 13:32:25 <dmcbride> narinder: are you still planning to release those with D1? 13:32:53 <dmcbride> chigang: I see problems with two of your scenarios on yardstick: 13:32:58 <dmcbride> os-odl_l2-nofeature-ha 13:32:58 <dmcbride> os-odl_l3-nofeature-ha 13:33:23 <narinder> dmcbride, deployment works some how some other issue https://build.opnfv.org/ci/job/joid-deploy-baremetal-daily-danube/53/ 13:33:49 <georgk> #info Georg Kunz 13:34:18 <dmcbride> narinder: when I checked yesterday, k8-nosdn-lb-noha was failing deploy 13:34:25 <dmcbride> narinder: is it passing now? 13:34:46 <narinder> check this one it is passed https://build.opnfv.org/ci/job/joid-deploy-baremetal-daily-danube/53/console 13:34:52 <narinder> i have not checked the latest 13:34:53 <chigang> dmcbride: yes, we are in troubleshooting, but not failed in every time. 13:35:09 <narinder> but looks like this is avialaiblity of lab issue 13:35:23 <trozet> i'm here btw..sorry late 13:35:59 <dmcbride> chigang: os-odl_l3-nofeature-ha had failed yardstick every time (0/10) when I looked late yesterday 13:36:13 <dmcbride> trozet: hi Tim, glad you could join 13:37:18 <dmcbride> trozet: or frankbrockners fdio scenarios seem to be declining on functest 13:37:37 <chigang> dmcbride: sorry, I thought you mentioned functest 13:37:41 <dmcbride> trozet: or frankbrockners any idea what's going on there 13:38:06 <dmcbride> chigang: no, I was referring to yardstick 13:38:14 <trozet> dmcbride: the issue is the fdio scenarios seem to fail on lfpod1 for testing, but pass in other environments 13:38:31 <chigang> dmcbride: OK, I will check it tomorrow. 13:38:48 <trozet> dmcbride: and if i login to the lfpod1 and test it manually it works, so we are not quite sure what the problem is yet. I have requested more debug logs from functest when a failure occurs so I can try to investigate 13:39:14 <dmcbride> chigang: os-odl_l3-nofeature-ha seems to be degrading over time on functest, if you look at the spreadsheet I sent yesterday 13:39:28 <trozet> sorry I should have bene using info 13:39:32 <trozet> let me fix it 13:39:51 <narinder> dmcbride, also k8 required to remove the functest and yardstick which is not valid. 13:39:53 <trozet> #info fdio functest seems to fail in lfpod1, but passes in other environments...investigating the cause 13:39:59 <trozet> #info for apex that is 13:41:17 <dmcbride> frankbrockners: are you aware of this issue that Tim has identified? 13:41:46 <trozet> dmcbride: yeah he is aware, we have some FDS engineers also trying to reproduce it 13:42:43 <dmcbride> #info dmcbride notes problems with compass scenarios os-odl_l2-nofeature-ha and os-odl_l3-nofeature-ha 13:43:05 <dmcbride> #info chigang will investigate 13:43:08 <trozet> #info Apex has os-nosdn-ovs-ha and os-odl_l3-ovs-ha working, adding it to daily for todays run 13:43:33 <chigang> dmcbride: as I mentioned before, there are some testcases failed sometimes, it is not too easy to fix. 13:43:43 <chigang> dmcbride: I will check them 13:43:46 <dmcbride> pma: are you available 13:43:59 <pma_> dmcbride> yep 13:44:05 <dmcbride> chigang: that's why you get paid the big bucks :) 13:44:34 <dmcbride> pma_: seems like we are seeing a lot of problems with Fuel 13:44:48 <pma_> we've issue w/ yardstick against all odl scenarios https://jira.opnfv.org/browse/YARDSTICK-603 13:45:24 <pma_> yeah, plus upstream bug https://bugs.launchpad.net/fuel/+bug/1676197 13:45:26 <dmcbride> #info dmcbride says that only one candidate scenario on Fuel was passing as of yesterday 13:45:57 <dmcbride> pma_: is that bug a root cause, or just one of a number of problems? 13:46:17 <pma_> dmcbride> it's blocker for all deployments 13:46:32 <dmcbride> #link https://bugs.launchpad.net/fuel/+bug/1676197 13:46:44 <pma_> dmcbride> we're working on it, fix is coming ... 13:46:47 <dmcbride> pma_: is this an issue you've told me about before? 13:47:19 <dmcbride> #info pma_ says that this is a blocker for all deployments ^^ 13:47:20 <pma_> dmcbride> no, it's new, related to the feature of distributed workes 13:48:37 <dmcbride> narinder: I checked again and k8-nosdn-lb-noha still does not appear to be passing deploy 13:49:47 <narinder> dmcbride, i am seeing something different r u seeing by color or looking for deployment status. Check this link https://build.opnfv.org/ci/view/joid/job/joid-deploy-baremetal-daily-danube/ 13:50:33 <narinder> if it is color then it might be scheduling issues as i was seeing multiple jobs were schedules but no executor 13:50:53 <narinder> so i have to abort few of the scenarios today 13:50:55 <dmcbride> narinder: I'm looking at this: https://build.opnfv.org/ci/job/joid-k8-nosdn-lb-noha-baremetal-daily-danube/ 13:51:31 <narinder> dmcbride, march 24th deployment got passed 13:51:51 <narinder> this is detailed deployment job https://build.opnfv.org/ci/view/joid/job/joid-deploy-baremetal-daily-danube/53/console 13:52:30 <dmcbride> narinder: ok - thanks 13:52:41 <narinder> dmcbride, also in this https://build.opnfv.org/ci/job/joid-k8-nosdn-lb-noha-baremetal-daily-danube/ deployment was passed 13:52:55 <narinder> see the first bullet which is blue color 13:54:07 <dmcbride> narinder: ok - got it 13:54:18 <trozet> #info Apex: os-odl_l3-feature-ha functest was fixed, I see the trend score moving upwards 13:54:21 <narinder> dmcbride, cool thanks 13:54:25 <dmcbride> narinder: I'm still figuring out how to read Jenkins 13:55:04 <narinder> dmcbride, it is confusing for me sometimes. Even though it should be straight forward but not some times. 13:56:07 <dmcbride> trozet: that is consistent with the spreadsheet 13:56:45 <dmcbride> trozet: 15 ==> 21 ==> 26 13:56:52 <trozet> dmcbride: from Apex side, i think our main focus right now is fixing FDIO and bgpvpn scenarios 13:57:54 <jose_lausuch> trozet: Nikolas is looking into bgpvpn I think 13:58:11 <dmcbride> trozet: I also see that yardstick is either not running or is failing on most Apex scenarios 13:58:18 <trozet> jose_lausuch: ok cool, was just looking to see if he was online to ping him :) I think he filed a bug that quagga wasnt coming up right 13:58:41 <trozet> dmcbride: i havent really looked at yardstick at all yet, been focused on functest 13:59:04 <trozet> radez: ping? 13:59:54 <radez> hey trozet 14:00:14 <dmcbride> #info narinder notes that dmcbride read the deploy data in Jenkins wrong for k8-nosdn-lb-noha 14:00:29 <dmcbride> #action dmcbride to update spreadsheet 14:01:20 <trozet> radez: can you look through apex yardstick results on the daily job and try to determine why they are not passing? 14:01:32 <trozet> radez: maybe start with os-nosdn-nofeature as that one is passing almost 100% functest 14:02:36 <radez> trozet: won't be able to do today, but I can put it on my list for first thing tomorrow 14:02:55 <trozet> radez: ok thanks 14:03:05 <trozet> #action radez to look into Apex yardstick failures in daily 14:03:26 <trozet> radez: while we got you here, want to give an update on OVN ha? 14:04:52 <dmcbride> trozet: is radez on another channel? 14:05:10 <trozet> dmcbride: he's in here 14:05:16 <dmcbride> trozet: I'm not seeing his input on opnfv-release 14:05:25 <trozet> dmcbride: oh, i see it 14:05:25 <dmcbride> trozet: or is he just lurking ;) 14:05:39 <frankbrockners> trozet: sorry - in a IETF meeting right now - Michal is investigating why L3 failed deployment - we did not really change anything 14:06:22 <dmcbride> frankbrockners: trozet thinks the fdio failures may be pod-specific 14:06:58 <trozet> frankbrockners: we can talk about that one in the meeting, i think we are more concerned about fdio odl_l2 noha failing functest 14:07:07 <trozet> frankbrockners: it passes in cengn lab 14:07:26 <trozet> frankbrockners, dmcbride: i'm going to kill the daily when i see the functest start for that scenario and go try to debug 14:08:03 <frankbrockners> trozet: yes - I asked jlinkes as well... - let's nail this in the FDS meeting 14:08:04 <dmcbride> frankbrockners: any other updates for us before we end the meeting? 14:08:09 <trozet> jose_lausuch: fyi we could use some more log collection when the healthcheck fails. Did you ever get that added to functest? Should we be running with DEBUG true? 14:08:43 <frankbrockners> trozet: could you also enable nosdn-fdio-noha in the pipeline... the nosdn-fdio-ha scenario makes progress, but some issues remain... 14:08:44 <radez> sry, was distracted for a min, we will have OVN non-HA for D1 14:08:55 <jose_lausuch> trozet: what kind of logs? from the deployment as such? 14:08:59 <trozet> frankbrockners: yes I will enable it 14:09:04 <radez> I'm not sure about OVN-HA yet still investigating it 14:09:06 <frankbrockners> trozet: thanks 14:09:09 <jose_lausuch> trozet: nova.log and so? or functest logs? 14:09:26 <dmcbride> jose_lausuch: that reminds me. I will bring this up in the TWG, but I think that it would be useful to display a timestamp for each scenario in the functest dashboard 14:09:33 <radez> trozet: in fact I think I'm going to pull out the senario file for ovn-ha and get the non-ha verified and merged before I continue with the ha stuff 14:09:38 <trozet> jose_lausuch: right, we see dhcp fail, and what we really need are some logs from neutron, and nova, and vpp. We also need a virsh dumpxml of the instance so we can verify it has a vhostuser port 14:09:56 <trozet> radez: that works, we can just claim support for the noha version and add it to the daily 14:10:04 <jose_lausuch> trozet: I started working on it before my 2 weeks off, I plan to continue this week 14:10:26 <jose_lausuch> dmcbride: you mean duration? 14:10:52 <dmcbride> jose_lausuch: time stamp for last run 14:12:14 <dmcbride> jose_lausuch: if I remember correctly, the data covers a 10 day window, but you have no idea if the most recent run was hours ago or days ago 14:12:15 <jose_lausuch> dmcbride: we have a link to the last run of you click on it 14:12:18 <radez> trozet: and fyi we'll have ovn-ha support no problem in E, the issue with HA appears to be puppet code discrepancies in the backports for tht and puppet-ooo 14:12:26 <jose_lausuch> dmcbride: ok, I get it now, yes we can add that maybe 14:12:54 <trozet> radez: sure if ovn wants HA then they can help us add it, we've done our part 14:13:08 <trozet> radez: s/we/you/ 14:13:20 <radez> ack 14:17:12 <dmcbride> no daily meeting tomorrow, since we will have the regular weekly release meeting 14:17:26 <dmcbride> so, the next daily meeting will be on Wednesday at the same time 14:18:04 <dmcbride> #info next daily release meeting on Wednesday at 6:30 a.m. Pacific 14:18:29 <dmcbride> #endmeeting