15:02:35 <dmcbride> #startmeeting OPNFV Colorado 1.0 daily release meeting
15:02:35 <collabot`> Meeting started Thu Sep  8 15:02:35 2016 UTC.  The chair is dmcbride. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:35 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:02:35 <collabot`> The meeting name has been set to 'opnfv_colorado_1_0_daily_release_meeting'
15:02:47 <dmcbride> #topic roll call
15:02:53 <dmcbride> #info David McBride
15:03:04 <jose_lausuch> irc only?
15:03:09 <jose_lausuch> #info Jose Lausuch
15:03:29 <dmcbride> jose_lausuch:  yes, IRC only
15:03:51 <jmorgan1> #info Jack Morgan (Pharos)
15:05:12 <trozet> #info Tim Rozet (Apex)
15:05:54 <frankbrockners> #info Frank Brockners
15:06:03 <fdegir> #info Fatih Degirmenci
15:06:04 <narindergupta> #info Narinder Gupta
15:07:34 <dmcbride> ok - thanks everyone for joining the meeting
15:08:13 <dmcbride> this (near) daily is in response to a request that came up during the infra meeting on Wednesday to initiate a daily meeting to discuss critical issues for the Col 1.0 release
15:08:53 <dmcbride> I'd like to focus on Colorado 1.0 (Sept 22) for this meeting and defer other release topics to the regular release meeting, or the TSC meeting
15:09:53 <morgan_orange> #info Morgan Richomme
15:10:06 <dmcbride> so, first up, I'd like to get feedback from installer PTLs, starting with Apex
15:10:08 <ChrisPriceAB> #info Chris Price (IRC only)
15:10:19 <dmcbride> #topic installer reports
15:10:42 <dmcbride> trozet: one thing I'm wondering about is the status of Functest for Apex
15:10:52 <jose_lausuch> ChrisPriceAB: lucky you, this meeting is irc only :)
15:10:57 <dmcbride> status for all scenarios is showing red
15:10:58 <trozet> dmcbride: which part?
15:11:08 <ChrisPriceAB> a rarity indeed jose_lausuch
15:11:29 <trozet> dmcbride: the red in jenkins just means not 100% pass
15:11:36 <dmcbride> #link http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html
15:12:06 <trozet> dmcbride: i don't know what these mean
15:12:09 <morgan_orange> trozet:  dmcbride means not 100% pass 4 times in a raw...
15:12:18 <jose_lausuch> I see a lot of "suns" though which is a good signal
15:12:31 <dmcbride> understood - but Apex seems to be the only one of the 4 installer pages that is all red
15:12:43 <trozet> dmcbride: i think the lower part of hte page is better to look at, the sun :)
15:12:44 <dmcbride> every other page shows at least 3 passing
15:13:23 <morgan_orange> copper has been fixed recently needs 4 successful runs
15:13:38 <trozet> dmcbride: so if you look at hte first scenario 'os-nosdn-nofeature-ha', it is all sunny except copper
15:13:47 <dmcbride> trozet: I'm not trying to beat up on Apex, but I'm wondering if there is something unique going on with Apex that separates these results from the other installers
15:14:19 <morgan_orange> dmcbride: recent fix on copper....first scenarios should turn green soon
15:14:36 <morgan_orange> moreover it is functest view, it is not the release criteria...
15:14:39 <trozet> dmcbride: i think we were the only ones trying to do copper in HA
15:14:51 <trozet> dmcbride: I found last weekend it doesnt work in HA correctly
15:15:09 <jose_lausuch> yes, some of them will turn green soon
15:15:11 <trozet> dmcbride: looked like congress bugs to me, so we recently disabled copper as an HA service (shut it down on 2/3 nodes)
15:15:20 <jose_lausuch> for others like sfc or bgpvpn maybe not so quick
15:16:15 <trozet> jose_lausuch, morgan_orange: what does the "Score" like 16/18 mean? and what does the score have to be for the red X to turn green?
15:16:52 <morgan_orange> trozet: https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
15:17:16 <morgan_orange> 16/18 means 2 runs are expected for copper so we will have 4 and 4 and a global scoring of 18/18
15:17:24 <jose_lausuch> I Was trying to find the link, thanks
15:17:29 <morgan_orange> we turn green when we have the max scoring
15:17:38 <morgan_orange> but as said
15:17:39 <dmcbride> trozet: so, is the issue Copper or Congress? Or, don't know, yet?
15:17:42 <morgan_orange> it is a Functest representation
15:17:45 <morgan_orange> to give feedback
15:17:53 <morgan_orange> it does not intend to be the scenario validation....
15:17:54 <trozet> dmcbride: issue is with congress code
15:18:04 <morgan_orange> yardstick has similar page
15:19:37 <trozet> jose_lausuch,  morgan_orange: so i guess the problem to me with this methodology is, if a functest run contains hundreds of tests, and just copper fails, it is considered a failure even though >95% of the tests pass
15:19:51 <trozet> jose_lausuch, morgan_orange: so if that happens 4 times its considered 0 points
15:19:52 <dmcbride> #info trozet reports that issues with Apex scenarios in functest tied to bugs in congress
15:20:11 <trozet> jose_lausuch, morgan_orange: even though the scenario is passing almost every test
15:20:19 <morgan_orange> trozet: yes but as it is a page to give feedback and not validate the scenario
15:21:06 <dmcbride> morgan_orange: ? if the requirement is to pass 4 consecutive times, how is that not validating the scenario?
15:21:06 <trozet> it would probably be a better method to not look at hte jenkins result, and instead look at overall pass fail rate of tests to determine if an overall functest is successful
15:21:26 <bryan_att> input on that; I did not plan to have Copper included in HA scenarios, as I have not tested that. I can look into the issues but at this point I would suggest to leave it out of those scenarios for now, unless it can be installed on just one server (not in HA mode).
15:21:31 <dmcbride> trozet: yes, that's what I've been doing
15:22:03 <morgan_orange> dmcbride: it is on the definition of consecutive times...we can say that we run it 4 times and have minor errors then validate the scenario
15:22:07 <trozet> dmcbride: i think the lower half of that page shows that type of granularity
15:22:08 <dmcbride> I'm looking at the functest scoring as a measure of scenario performance
15:22:34 <morgan_orange> dmcbride: why Functest is not the only test project
15:22:42 <dmcbride> trozet: those are the ratios shown in the column next to the red and green balls
15:23:01 <trozet> dmcbride: i'm saying those are not as accurate as if you look where hte sun/clouds are
15:23:12 <trozet> dmcbride: so if you look at os-nosdn-nofeature-ha everything is sunny, except copper
15:23:20 <dmcbride> trozet: still, it stood out to me that none of the scenarios on Apex were green, while all of the other scenarios have at least 3 green
15:23:20 <trozet> dmcbride: which is pretty good
15:23:29 <dmcbride> all of the other installers
15:23:58 <trozet> dmcbride: yeah to me it is a misrepresentation.  For example, if only Apex supports doctor and the doctor test fails, we get a red X, vs other installers that dont run the doctor test
15:24:20 <morgan_orange> yes that is why it is a functest report not the scenario reporting...
15:24:31 <jose_lausuch> morgan_orange: 3rd time  :)
15:24:34 <trozet> dmcbride: hence why i suggest looking at the lower part of hte page
15:24:40 <morgan_orange> the more feature tests you will have, the most difficult it will be to get the top score
15:24:53 <bryan_att> pain=gain
15:25:15 <dmcbride> bryan_att: thanks for joining
15:25:39 <bryan_att> sent a note a little earlier about the Copper/HA question. Not sure you saw that.
15:25:53 <dmcbride> bryan_att: Tim reports that congress issues are causing failures in functest for copper on Apex
15:25:57 <jose_lausuch> I insist, this provides feedback to the test owners but I don't think it should be used to decide scenario validation, just as a guide
15:26:08 <morgan_orange> if you consider yardstick approach (kubi tell me if I am wrong) you get green if the test was launched (whatever the results are)
15:26:34 <morgan_orange> if we consider this page 1 months ago, all the scores were low
15:26:38 <dmcbride> bryan_att: was your note via email?
15:26:44 <bryan_att> dmcbride: well aware, see the note I put into IRC. No expectation of HA scenario support for Copper in Colorado. That was an Apex experiment that I didn't have time to validate.
15:26:45 <morgan_orange> we can see that they are almost all good now
15:26:59 <jose_lausuch> dmcbride: I think we can use the scoring as a guide, but without sticking 100% to that
15:27:03 <morgan_orange> we can move from a green/red status to a gauge
15:27:26 <trozet> usually in our HA scenarios - we want all openstack services in HA
15:27:33 <bryan_att> I have no specific evidence that the Congress service works in a load-balanced environment, and no lab environment in which to test it
15:27:42 <trozet> so we should document congress issues in HA and doesn't run HA in OPNFV
15:27:50 <bryan_att> trozet: I understand, are all services held to that expectation?
15:27:51 <dmcbride> morgan_orange: I chatted with jose_lausuch recently about that.  Would be good to be able to plot scoring over time for each scenario.
15:27:55 <trozet> bryan_att: yeah
15:28:09 <morgan_orange> dmcbride: dev is frozen...
15:28:21 <bryan_att> trozet: as noted I am fine with Copper being left out of HA scenarios.
15:28:28 <jose_lausuch> dmcbride: yes, I think we can take it for D-river, it would be nice to have a simple graph
15:28:33 <bryan_att> I will get to it in D release
15:28:38 <jose_lausuch> nobody has time now to implement that
15:28:39 <dmcbride> trozet: and bryan_att can we disable those tests on Apex if we do not intent to deliver that functionality?
15:28:44 <trozet> bryan_att: i think it is fine to include it in non-ha, and just document the limitation.  It's good to have it any capacity rather than not have it
15:29:08 <bryan_att> I am not suggesting to pull it entirely - only for HA scenarios
15:29:11 <trozet> dmcbride: it's an option if we continue to see the test fail and determine it is not Apex (we have already disabled HA for it)
15:29:20 <trozet> right bryan_att
15:29:20 <narindergupta> dmcbride, i have to goto another meeting.
15:29:25 <bryan_att> dmcbride: the tests are already disabled
15:29:30 <dmcbride> narindergupta: any critical issues to report for joid?
15:30:09 <narindergupta> dmcbride, no last week CI was down but its up now
15:30:20 <narindergupta> and colorado stable releases are passing
15:30:32 <morgan_orange> narinder lxd ha and noha shall turn green as well (at least no more Rally errors)
15:30:32 <bryan_att> btw, there is one potential side-effect of the HA limitation for Congress - it may affect Doctor tests in HA since they are using it I think
15:30:58 <narindergupta> morgan_orange, yes it is
15:31:12 <narindergupta> morgan_orange, only problem is yardstick now
15:31:22 <dmcbride> #info narindergupta reports that colorado stable releases are passing on joid
15:31:49 <dmcbride> someone asked why I was only looking at functest
15:32:18 <dmcbride> I wanted to also look at yardstick, but there is apparently no colorado-specific dashboard for yardstick
15:33:13 <dmcbride> ok - looks like we're out of time
15:33:19 <morgan_orange> http://testresults.opnfv.org/reporting/yardstick/release/stable/colorado/index-status-apex.html
15:33:29 <dmcbride> we will resume same time, same channel on Friday
15:34:18 <dmcbride> morgan_orange: thanks
15:34:28 <dmcbride> morgan_orange: image links appear to be broken
15:34:40 <morgan_orange> dmcbride: they have an issue with the images due to the path in their template
15:34:45 <dmcbride> not seeing the red and green balls
15:34:59 <jose_lausuch> need to drop out sorry
15:35:09 <morgan_orange> they use the same tempalte but some results are in master and the seond one in stable/colorado
15:35:12 <dmcbride> jose_lausuch: thanks - we're done for today
15:35:22 <morgan_orange> I reported it to yardstick
15:35:29 <morgan_orange> but you can see the score..
15:35:54 <morgan_orange> I can fix it temprarily but tomorrow it will be overwritten if they do not fix the path issue in the template
15:36:02 <dmcbride> morgan_orange: but I like the red and green balls - they're perty ;)
15:36:24 <morgan_orange> but it can be misleading...red with a high score is almost green...
15:36:41 <dmcbride> morgan_orange: yes - I get it
15:37:34 <dmcbride> morgan_orange: looks like much fewer iterations for yardstick as compared to functest
15:38:15 <dmcbride> #endmeeting