13:29:58 <dmcbride> #startmeeting OPNFV release daily 13:29:58 <collabot`> Meeting started Fri Mar 17 13:29:58 2017 UTC. The chair is dmcbride. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:29:58 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic. 13:29:58 <collabot`> The meeting name has been set to 'opnfv_release_daily' 13:30:10 <dmcbride> #topic roll call 13:30:16 <dmcbride> #info David McBride 13:30:23 <radez> #info r-mibu 13:30:26 <radez> #info radez 13:30:38 <radez> sry r-mibu my mistake 13:31:08 <sergmelikyan> #info Serg Melikyan 13:32:49 <dmcbride> narindergupta: can you join the release meeting? 13:33:30 <dmcbride> radez: do you know why we are seeing all of these deployment failures? 13:34:52 <dmcbride> sergmelikyan: any progress on the list of bugs that you sent me? 13:34:52 <radez> dmcbride: nope, is there one in particular you want me to look at and investigate? 13:35:23 <radez> dmcbride: I've been focused on development outside of ci recently so I'm not super familiar with it 13:35:44 <dmcbride> radez: if I'm reading the jenkins results correctly, there have been no successes for almost two weeks 13:35:53 <radez> link? 13:35:59 <dmcbride> so, it seems to be a systemic issue 13:36:24 <dmcbride> https://build.opnfv.org/ci/view/OPNFV%20Platform%20CI/builds 13:37:16 <sergmelikyan> dmcbride: yeah, it's increased by another issue with Yardstick https://lists.opnfv.org/pipermail/opnfv-tech-discuss/2017-March/015604.html 13:37:25 <radez> dmcbride: is it related to morgan's reply to your email yestertday? 13:37:49 <dmcbride> radez: that is the link that morgan sent yesterday in his email 13:38:01 <dmcbride> radez: so, yes 13:38:28 <sergmelikyan> pma: do you have more detailed update on this? 13:39:57 <dmcbride> chigang: can you join the meeting on IRC? 13:40:17 <chigang> dmcbride: now? 13:40:32 <chigang> dmcbride: yes, I can 13:40:45 <dmcbride> chigang: yes, IRC only 13:41:17 <dmcbride> chigang: I'm trying to understand why all of the deployments seem to be failing 13:42:09 <pma> regarding Fuel, we've some bugs in upstream for now 13:42:36 <pma> https://bugs.launchpad.net/fuel/+bug/1672969 13:42:45 <pma> https://bugs.launchpad.net/fuel/+bug/1672683 13:43:44 <dmcbride> ok, that explains Fuel, but why are Apex, Compass, and Joid deployments failing? 13:45:02 <dmcbride> radez: do you have blocking OOO bugs that would explain this? 13:45:35 <dmcbride> narindergupta: why are Joid deployments failing? 13:45:46 <radez> dmcbride: no ou blocking bugs have been around features. Triple-o has been pretty solid for us this release 13:45:51 <radez> *our blocking 13:46:04 <radez> I'm looking at logs to see if there is a pattern to ours or 13:46:11 <radez> if it's just working through details 13:46:40 <dmcbride> radez: do you see anything that's passing deployment? Looks like all failures to me. 13:46:40 <Bob_> #info Bob Monkman 13:46:59 <chigang> dmcbride: regarding to compass failed, because the environment for os version in danube branch is not correct, the patch has been merged https://gerrit.opnfv.org/gerrit/#/c/30659/ 13:47:01 <dmcbride> hello Bob Monkman, thanks for joining 13:47:38 <radez> dmcbride: looks like the last success we had was March 8th, I'm not sure why we haven't run since the 12th 13:47:45 <radez> https://build.opnfv.org/ci/job/apex-daily-master/423/ 13:47:47 <Bob_> dmcbride- sure thing sorry I am late 13:47:51 <dmcbride> chigang: can you give me more details? What do you mean by the "environment"? 13:47:55 <radez> that's our march 8th run, things look good there 13:48:13 <radez> from a deployment perspective atleast 13:48:55 <radez> dmcbride: oh, tim just told me that we have our daily fixed and should start to get results again today 13:49:31 <radez> dmcbride: sry I've been looking at master... here's danube 13:49:32 <radez> https://build.opnfv.org/ci/job/apex-daily-master/423/ 13:49:39 <radez> https://build.opnfv.org/ci/view/apex/job/apex-daily-danube/ 13:50:07 <radez> dmcbride: so build and deploy succedded and functest is running now 13:50:21 <dmcbride> well, at least the functest dashboard is starting to populate 13:50:23 <dmcbride> http://testresults.opnfv.org/reporting/functest/release/danube/index-status-joid.html 13:50:49 <chigang> dmcbride: OS version in each host in danube is set to "Ubuntu trusty", it should be "Ubuntu Xenial". it cause wrong OS version is provisioning in Danbue. 13:50:58 <bh526r> Hi, a quick question regarding official URL of Danube release: is it https://www.opnfv.org/danube? 13:51:22 <bh526r> Or is it https://www.opnfv.org/software? 13:51:50 <dmcbride> rpaik: are you available? 13:52:48 <dmcbride> chigang: do you have a plan to get that resolved? 13:52:57 <chigang> dmcbride: it has been fixed. it will take long time to verify all of the scenrios. and it seems no-sdn-no-feature works now. 13:53:11 <bh526r> The reason I am asking is that we may refer to the URL of official Danube release in our documentation. 13:53:50 <dmcbride> bh526r: yes - understood. I'm not sure of the answer. I was hoping Ray was online to respond to that. 13:54:03 <bh526r> I see, thank you David 13:54:18 <bh526r> I will ping Ray offline then 13:54:51 <dmcbride> guys, am I missing something when I look at that Jenkins dashboard? 13:55:05 <dmcbride> it looks like nothing but doom and gloom for over a week. 13:55:24 <dmcbride> yet, the Bitergia data, and other indicators seem to contradict that. 13:55:45 <dmcbride> So, it makes me wonder if I'm misunderstanding what the Jenkins dashboard is reporting. 13:58:01 <dmcbride> morgan_orange: that Jenkins link that you sent to me, yesterday seems to indicate that there have been no successful deployments for over a week 13:58:13 <dmcbride> morgan_orange: am I interpreting that correctly? 13:58:36 <dmcbride> narindergupta: why is Joid failing deployments? 13:58:52 <morgan_orange> depends on the installer/version 13:59:04 <dmcbride> narindergupta: why am I seeing no data for Joid scenarios on the functest dashboard for Danube? 13:59:28 <morgan_orange> regarding danube: we got 1 run today for Apex and Compass. Joid scenario since 2 days 13:59:58 <morgan_orange> dmcbride: joid is running well lxd and nosdn scenarios in ha and noha http://testresults.opnfv.org/reporting/functest/release/danube/index-status-joid.html 14:00:16 <morgan_orange> issue with odl scenario (like in master) 14:01:17 <morgan_orange> apex: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-apex.html 14:01:20 <dmcbride> morgan_orange: for example, this one: https://build.opnfv.org/ci/job/joid-os-odl_l2-nofeature-ha-baremetal-daily-danube/ 14:01:21 <morgan_orange> 1 run on nosdn scenario 14:01:42 <dmcbride> jenkins says that it has been broken "for a long time" 14:01:43 <morgan_orange> fd scenario from Cisco labs (it exmplains the stange trend lines ..not launched by OPNFV CI) 14:02:35 <morgan_orange> dmcbride: yes odl scenario is broken fonr a long time 14:02:58 <dmcbride> morgan_orange: is that consistent for all odl scenarios? 14:03:10 <morgan_orange> odl joid scenarios yes 14:03:14 <dmcbride> morgan_orange: is that an ODL bug? 14:03:43 <morgan_orange> probably not (working with other installer), it should be a joid charm issue 14:03:54 <morgan_orange> odl integration issue with joid 14:04:12 <dmcbride> just with joid, not other installers? 14:04:29 <pma> https://gerrit.opnfv.org/gerrit/30341 for Fuel 14:04:32 <dmcbride> narindergupta: ?? 14:05:16 <dmcbride> phrobb: are you aware of an ODL integration issue with Joid? 14:05:51 <morgan_orange> pma: I planned to merge, just wait for other +2 from functest core especially juha who deals with temepst/rally topics 14:06:36 <pma> morgan_orange> Ok, got it 14:06:54 <dmcbride> morgan_orange: you mentioned some recent successes. Why don't those appear on that Jenkins dashboard. It seems to show no successes until 9 days ago. 14:08:05 <morgan_orange> dmcbride: ? we see success on jenkins 14:08:28 <morgan_orange> see joid/nosdn https://build.opnfv.org/ci/view/functest/job/functest-joid-baremetal-daily-danube/10/ 14:08:29 <morgan_orange> doen today 14:08:36 <dmcbride> I'm looking at the link that you sent, yesterday: https://build.opnfv.org/ci/view/OPNFV%20Platform%20CI/builds 14:10:03 <morgan_orange> here you have incomplete runs - deploy can be OK, but functest and/or yardstick nto completed 14:10:42 <morgan_orange> (yellow balls) 14:10:56 <dmcbride> ah - ok 14:11:13 <dmcbride> so, if the ball is yellow, then it passed the deploy but failed a test? 14:11:24 <morgan_orange> yes 14:11:34 <morgan_orange> when everything is red, deployments fail 14:11:47 <morgan_orange> when yellow, at least 1 test is FAIL 14:11:54 <morgan_orange> when blue everythign is good 14:12:02 <morgan_orange> assuming that test = Functest + Yardstick 14:12:08 <morgan_orange> and Functest = somm of tests 14:12:33 <morgan_orange> we may have 1 single test from tempest we will set red..which is very strict but the best way to give feedback 14:15:14 <dmcbride> morgan_orange: ok - thanks for the clarification. 14:15:26 <morgan_orange> so after MS3, it shoudl be either yellow or blue :) 14:16:15 <dmcbride> morgan_orange: Good tip. I will remember that for Euphrates. 14:17:08 <dmcbride> ok - we are well over time for this meeting. Thanks for attending. The next daily will be on Monday at 6:30 am PST 14:18:31 <dmcbride> #endmeeting