13:29:58 <dmcbride> #startmeeting OPNFV release daily
13:29:58 <collabot`> Meeting started Fri Mar 17 13:29:58 2017 UTC.  The chair is dmcbride. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:29:58 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
13:29:58 <collabot`> The meeting name has been set to 'opnfv_release_daily'
13:30:10 <dmcbride> #topic roll call
13:30:16 <dmcbride> #info David McBride
13:30:23 <radez> #info r-mibu
13:30:26 <radez> #info radez
13:30:38 <radez> sry r-mibu my mistake
13:31:08 <sergmelikyan> #info Serg Melikyan
13:32:49 <dmcbride> narindergupta: can you join the release meeting?
13:33:30 <dmcbride> radez: do you know why we are seeing all of these deployment failures?
13:34:52 <dmcbride> sergmelikyan: any progress on the list of bugs that you sent me?
13:34:52 <radez> dmcbride: nope, is there one in particular you want me to look at and investigate?
13:35:23 <radez> dmcbride: I've been focused on development outside of ci recently so I'm not super familiar with it
13:35:44 <dmcbride> radez: if I'm reading the jenkins results correctly, there have been no successes for almost two weeks
13:35:53 <radez> link?
13:35:59 <dmcbride> so, it seems to be a systemic issue
13:36:24 <dmcbride> https://build.opnfv.org/ci/view/OPNFV%20Platform%20CI/builds
13:37:16 <sergmelikyan> dmcbride: yeah, it's increased by another issue with Yardstick https://lists.opnfv.org/pipermail/opnfv-tech-discuss/2017-March/015604.html
13:37:25 <radez> dmcbride: is it related to morgan's reply to your email yestertday?
13:37:49 <dmcbride> radez: that is the link that morgan sent yesterday in his email
13:38:01 <dmcbride> radez: so, yes
13:38:28 <sergmelikyan> pma: do you have more detailed update on this?
13:39:57 <dmcbride> chigang: can you join the meeting on IRC?
13:40:17 <chigang> dmcbride: now?
13:40:32 <chigang> dmcbride: yes, I can
13:40:45 <dmcbride> chigang: yes, IRC only
13:41:17 <dmcbride> chigang: I'm trying to understand why all of the deployments seem to be failing
13:42:09 <pma> regarding Fuel, we've some bugs in upstream for now
13:42:36 <pma> https://bugs.launchpad.net/fuel/+bug/1672969
13:42:45 <pma> https://bugs.launchpad.net/fuel/+bug/1672683
13:43:44 <dmcbride> ok, that explains Fuel, but why are Apex, Compass, and Joid deployments failing?
13:45:02 <dmcbride> radez: do you have blocking OOO bugs that would explain this?
13:45:35 <dmcbride> narindergupta: why are Joid deployments failing?
13:45:46 <radez> dmcbride: no ou blocking bugs have been around features. Triple-o has been pretty solid for us this release
13:45:51 <radez> *our blocking
13:46:04 <radez> I'm looking at logs to see if there is a pattern to ours or
13:46:11 <radez> if it's just working through details
13:46:40 <dmcbride> radez: do you see anything that's passing deployment?  Looks like all failures to me.
13:46:40 <Bob_> #info Bob Monkman
13:46:59 <chigang> dmcbride: regarding to compass failed, because the environment for os version in danube branch is not correct, the patch has been merged https://gerrit.opnfv.org/gerrit/#/c/30659/
13:47:01 <dmcbride> hello Bob Monkman, thanks for joining
13:47:38 <radez> dmcbride: looks like the last success we had was March 8th, I'm not sure why we haven't run since the 12th
13:47:45 <radez> https://build.opnfv.org/ci/job/apex-daily-master/423/
13:47:47 <Bob_> dmcbride- sure thing sorry I am late
13:47:51 <dmcbride> chigang: can you give me more details?  What do you mean by the "environment"?
13:47:55 <radez> that's our march 8th run, things look good there
13:48:13 <radez> from a deployment perspective atleast
13:48:55 <radez> dmcbride: oh, tim just told me that we have our daily fixed and should start to get results again today
13:49:31 <radez> dmcbride: sry I've been looking at master... here's danube
13:49:32 <radez> https://build.opnfv.org/ci/job/apex-daily-master/423/
13:49:39 <radez> https://build.opnfv.org/ci/view/apex/job/apex-daily-danube/
13:50:07 <radez> dmcbride: so build and deploy succedded and functest is running now
13:50:21 <dmcbride> well, at least the functest dashboard is starting to populate
13:50:23 <dmcbride> http://testresults.opnfv.org/reporting/functest/release/danube/index-status-joid.html
13:50:49 <chigang> dmcbride: OS version in each host in danube  is set to "Ubuntu trusty", it should be "Ubuntu Xenial". it cause wrong OS version is provisioning in Danbue.
13:50:58 <bh526r> Hi, a quick question regarding official URL of Danube release: is it https://www.opnfv.org/danube?
13:51:22 <bh526r> Or is it https://www.opnfv.org/software?
13:51:50 <dmcbride> rpaik: are you available?
13:52:48 <dmcbride> chigang: do you have a plan to get that resolved?
13:52:57 <chigang> dmcbride: it has been fixed.  it will take long time to verify all of the scenrios. and it seems no-sdn-no-feature works now.
13:53:11 <bh526r> The reason I am asking is that we may refer to the URL of official Danube release in our documentation.
13:53:50 <dmcbride> bh526r: yes - understood.  I'm not sure of the answer.  I was hoping Ray was online to respond to that.
13:54:03 <bh526r> I see, thank you David
13:54:18 <bh526r> I will ping Ray offline then
13:54:51 <dmcbride> guys, am I missing something when I look at that Jenkins dashboard?
13:55:05 <dmcbride> it looks like nothing but doom and gloom for over a week.
13:55:24 <dmcbride> yet, the Bitergia data, and other indicators seem to contradict that.
13:55:45 <dmcbride> So, it makes me wonder if I'm misunderstanding what the Jenkins dashboard is reporting.
13:58:01 <dmcbride> morgan_orange: that Jenkins link that you sent to me, yesterday seems to indicate that there have been no successful deployments for over a week
13:58:13 <dmcbride> morgan_orange: am I interpreting that correctly?
13:58:36 <dmcbride> narindergupta: why is Joid failing deployments?
13:58:52 <morgan_orange> depends on the installer/version
13:59:04 <dmcbride> narindergupta: why am I seeing no data for Joid scenarios on the functest dashboard for Danube?
13:59:28 <morgan_orange> regarding danube: we got 1 run today for Apex and Compass. Joid scenario since 2 days
13:59:58 <morgan_orange> dmcbride: joid is running well lxd and nosdn scenarios in ha and noha http://testresults.opnfv.org/reporting/functest/release/danube/index-status-joid.html
14:00:16 <morgan_orange> issue with odl scenario (like in master)
14:01:17 <morgan_orange> apex: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-apex.html
14:01:20 <dmcbride> morgan_orange: for example, this one:  https://build.opnfv.org/ci/job/joid-os-odl_l2-nofeature-ha-baremetal-daily-danube/
14:01:21 <morgan_orange> 1 run on nosdn scenario
14:01:42 <dmcbride> jenkins says that it has been broken "for a long time"
14:01:43 <morgan_orange> fd scenario from Cisco labs (it exmplains the stange trend lines ..not launched by OPNFV CI)
14:02:35 <morgan_orange> dmcbride: yes odl scenario is broken fonr a long time
14:02:58 <dmcbride> morgan_orange: is that consistent for all odl scenarios?
14:03:10 <morgan_orange> odl joid scenarios yes
14:03:14 <dmcbride> morgan_orange: is that an ODL bug?
14:03:43 <morgan_orange> probably not (working with other installer), it should be a joid charm issue
14:03:54 <morgan_orange> odl integration issue with joid
14:04:12 <dmcbride> just with joid, not other installers?
14:04:29 <pma> https://gerrit.opnfv.org/gerrit/30341  for Fuel
14:04:32 <dmcbride> narindergupta: ??
14:05:16 <dmcbride> phrobb: are you aware of an ODL integration issue with Joid?
14:05:51 <morgan_orange> pma: I planned to merge, just wait for other +2 from functest core especially juha who deals with temepst/rally topics
14:06:36 <pma> morgan_orange> Ok, got it
14:06:54 <dmcbride> morgan_orange: you mentioned some recent successes.  Why don't those appear on that Jenkins dashboard.  It seems to show no successes until 9 days ago.
14:08:05 <morgan_orange> dmcbride: ? we see success on jenkins
14:08:28 <morgan_orange> see joid/nosdn https://build.opnfv.org/ci/view/functest/job/functest-joid-baremetal-daily-danube/10/
14:08:29 <morgan_orange> doen today
14:08:36 <dmcbride> I'm looking at the link that you sent, yesterday:  https://build.opnfv.org/ci/view/OPNFV%20Platform%20CI/builds
14:10:03 <morgan_orange> here you have incomplete runs - deploy can be OK, but functest and/or yardstick nto completed
14:10:42 <morgan_orange> (yellow balls)
14:10:56 <dmcbride> ah - ok
14:11:13 <dmcbride> so, if the ball is yellow, then it passed the deploy but failed a test?
14:11:24 <morgan_orange> yes
14:11:34 <morgan_orange> when everything is red, deployments fail
14:11:47 <morgan_orange> when yellow, at least 1 test is FAIL
14:11:54 <morgan_orange> when blue everythign is good
14:12:02 <morgan_orange> assuming that test = Functest + Yardstick
14:12:08 <morgan_orange> and Functest = somm of tests
14:12:33 <morgan_orange> we may have 1 single test from tempest we will set red..which is very strict but the best way to give feedback
14:15:14 <dmcbride> morgan_orange: ok - thanks for the clarification.
14:15:26 <morgan_orange> so after MS3, it shoudl be either yellow or blue :)
14:16:15 <dmcbride> morgan_orange: Good tip.  I will remember that for Euphrates.
14:17:08 <dmcbride> ok - we are well over time for this meeting.  Thanks for attending.   The next daily will be on Monday at 6:30 am PST
14:18:31 <dmcbride> #endmeeting