15:02:35 <dmcbride> #startmeeting OPNFV Colorado 1.0 daily release meeting 15:02:35 <collabot`> Meeting started Thu Sep 8 15:02:35 2016 UTC. The chair is dmcbride. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:02:35 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic. 15:02:35 <collabot`> The meeting name has been set to 'opnfv_colorado_1_0_daily_release_meeting' 15:02:47 <dmcbride> #topic roll call 15:02:53 <dmcbride> #info David McBride 15:03:04 <jose_lausuch> irc only? 15:03:09 <jose_lausuch> #info Jose Lausuch 15:03:29 <dmcbride> jose_lausuch: yes, IRC only 15:03:51 <jmorgan1> #info Jack Morgan (Pharos) 15:05:12 <trozet> #info Tim Rozet (Apex) 15:05:54 <frankbrockners> #info Frank Brockners 15:06:03 <fdegir> #info Fatih Degirmenci 15:06:04 <narindergupta> #info Narinder Gupta 15:07:34 <dmcbride> ok - thanks everyone for joining the meeting 15:08:13 <dmcbride> this (near) daily is in response to a request that came up during the infra meeting on Wednesday to initiate a daily meeting to discuss critical issues for the Col 1.0 release 15:08:53 <dmcbride> I'd like to focus on Colorado 1.0 (Sept 22) for this meeting and defer other release topics to the regular release meeting, or the TSC meeting 15:09:53 <morgan_orange> #info Morgan Richomme 15:10:06 <dmcbride> so, first up, I'd like to get feedback from installer PTLs, starting with Apex 15:10:08 <ChrisPriceAB> #info Chris Price (IRC only) 15:10:19 <dmcbride> #topic installer reports 15:10:42 <dmcbride> trozet: one thing I'm wondering about is the status of Functest for Apex 15:10:52 <jose_lausuch> ChrisPriceAB: lucky you, this meeting is irc only :) 15:10:57 <dmcbride> status for all scenarios is showing red 15:10:58 <trozet> dmcbride: which part? 15:11:08 <ChrisPriceAB> a rarity indeed jose_lausuch 15:11:29 <trozet> dmcbride: the red in jenkins just means not 100% pass 15:11:36 <dmcbride> #link http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html 15:12:06 <trozet> dmcbride: i don't know what these mean 15:12:09 <morgan_orange> trozet: dmcbride means not 100% pass 4 times in a raw... 15:12:18 <jose_lausuch> I see a lot of "suns" though which is a good signal 15:12:31 <dmcbride> understood - but Apex seems to be the only one of the 4 installer pages that is all red 15:12:43 <trozet> dmcbride: i think the lower part of hte page is better to look at, the sun :) 15:12:44 <dmcbride> every other page shows at least 3 passing 15:13:23 <morgan_orange> copper has been fixed recently needs 4 successful runs 15:13:38 <trozet> dmcbride: so if you look at hte first scenario 'os-nosdn-nofeature-ha', it is all sunny except copper 15:13:47 <dmcbride> trozet: I'm not trying to beat up on Apex, but I'm wondering if there is something unique going on with Apex that separates these results from the other installers 15:14:19 <morgan_orange> dmcbride: recent fix on copper....first scenarios should turn green soon 15:14:36 <morgan_orange> moreover it is functest view, it is not the release criteria... 15:14:39 <trozet> dmcbride: i think we were the only ones trying to do copper in HA 15:14:51 <trozet> dmcbride: I found last weekend it doesnt work in HA correctly 15:15:09 <jose_lausuch> yes, some of them will turn green soon 15:15:11 <trozet> dmcbride: looked like congress bugs to me, so we recently disabled copper as an HA service (shut it down on 2/3 nodes) 15:15:20 <jose_lausuch> for others like sfc or bgpvpn maybe not so quick 15:16:15 <trozet> jose_lausuch, morgan_orange: what does the "Score" like 16/18 mean? and what does the score have to be for the red X to turn green? 15:16:52 <morgan_orange> trozet: https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617 15:17:16 <morgan_orange> 16/18 means 2 runs are expected for copper so we will have 4 and 4 and a global scoring of 18/18 15:17:24 <jose_lausuch> I Was trying to find the link, thanks 15:17:29 <morgan_orange> we turn green when we have the max scoring 15:17:38 <morgan_orange> but as said 15:17:39 <dmcbride> trozet: so, is the issue Copper or Congress? Or, don't know, yet? 15:17:42 <morgan_orange> it is a Functest representation 15:17:45 <morgan_orange> to give feedback 15:17:53 <morgan_orange> it does not intend to be the scenario validation.... 15:17:54 <trozet> dmcbride: issue is with congress code 15:18:04 <morgan_orange> yardstick has similar page 15:19:37 <trozet> jose_lausuch, morgan_orange: so i guess the problem to me with this methodology is, if a functest run contains hundreds of tests, and just copper fails, it is considered a failure even though >95% of the tests pass 15:19:51 <trozet> jose_lausuch, morgan_orange: so if that happens 4 times its considered 0 points 15:19:52 <dmcbride> #info trozet reports that issues with Apex scenarios in functest tied to bugs in congress 15:20:11 <trozet> jose_lausuch, morgan_orange: even though the scenario is passing almost every test 15:20:19 <morgan_orange> trozet: yes but as it is a page to give feedback and not validate the scenario 15:21:06 <dmcbride> morgan_orange: ? if the requirement is to pass 4 consecutive times, how is that not validating the scenario? 15:21:06 <trozet> it would probably be a better method to not look at hte jenkins result, and instead look at overall pass fail rate of tests to determine if an overall functest is successful 15:21:26 <bryan_att> input on that; I did not plan to have Copper included in HA scenarios, as I have not tested that. I can look into the issues but at this point I would suggest to leave it out of those scenarios for now, unless it can be installed on just one server (not in HA mode). 15:21:31 <dmcbride> trozet: yes, that's what I've been doing 15:22:03 <morgan_orange> dmcbride: it is on the definition of consecutive times...we can say that we run it 4 times and have minor errors then validate the scenario 15:22:07 <trozet> dmcbride: i think the lower half of that page shows that type of granularity 15:22:08 <dmcbride> I'm looking at the functest scoring as a measure of scenario performance 15:22:34 <morgan_orange> dmcbride: why Functest is not the only test project 15:22:42 <dmcbride> trozet: those are the ratios shown in the column next to the red and green balls 15:23:01 <trozet> dmcbride: i'm saying those are not as accurate as if you look where hte sun/clouds are 15:23:12 <trozet> dmcbride: so if you look at os-nosdn-nofeature-ha everything is sunny, except copper 15:23:20 <dmcbride> trozet: still, it stood out to me that none of the scenarios on Apex were green, while all of the other scenarios have at least 3 green 15:23:20 <trozet> dmcbride: which is pretty good 15:23:29 <dmcbride> all of the other installers 15:23:58 <trozet> dmcbride: yeah to me it is a misrepresentation. For example, if only Apex supports doctor and the doctor test fails, we get a red X, vs other installers that dont run the doctor test 15:24:20 <morgan_orange> yes that is why it is a functest report not the scenario reporting... 15:24:31 <jose_lausuch> morgan_orange: 3rd time :) 15:24:34 <trozet> dmcbride: hence why i suggest looking at the lower part of hte page 15:24:40 <morgan_orange> the more feature tests you will have, the most difficult it will be to get the top score 15:24:53 <bryan_att> pain=gain 15:25:15 <dmcbride> bryan_att: thanks for joining 15:25:39 <bryan_att> sent a note a little earlier about the Copper/HA question. Not sure you saw that. 15:25:53 <dmcbride> bryan_att: Tim reports that congress issues are causing failures in functest for copper on Apex 15:25:57 <jose_lausuch> I insist, this provides feedback to the test owners but I don't think it should be used to decide scenario validation, just as a guide 15:26:08 <morgan_orange> if you consider yardstick approach (kubi tell me if I am wrong) you get green if the test was launched (whatever the results are) 15:26:34 <morgan_orange> if we consider this page 1 months ago, all the scores were low 15:26:38 <dmcbride> bryan_att: was your note via email? 15:26:44 <bryan_att> dmcbride: well aware, see the note I put into IRC. No expectation of HA scenario support for Copper in Colorado. That was an Apex experiment that I didn't have time to validate. 15:26:45 <morgan_orange> we can see that they are almost all good now 15:26:59 <jose_lausuch> dmcbride: I think we can use the scoring as a guide, but without sticking 100% to that 15:27:03 <morgan_orange> we can move from a green/red status to a gauge 15:27:26 <trozet> usually in our HA scenarios - we want all openstack services in HA 15:27:33 <bryan_att> I have no specific evidence that the Congress service works in a load-balanced environment, and no lab environment in which to test it 15:27:42 <trozet> so we should document congress issues in HA and doesn't run HA in OPNFV 15:27:50 <bryan_att> trozet: I understand, are all services held to that expectation? 15:27:51 <dmcbride> morgan_orange: I chatted with jose_lausuch recently about that. Would be good to be able to plot scoring over time for each scenario. 15:27:55 <trozet> bryan_att: yeah 15:28:09 <morgan_orange> dmcbride: dev is frozen... 15:28:21 <bryan_att> trozet: as noted I am fine with Copper being left out of HA scenarios. 15:28:28 <jose_lausuch> dmcbride: yes, I think we can take it for D-river, it would be nice to have a simple graph 15:28:33 <bryan_att> I will get to it in D release 15:28:38 <jose_lausuch> nobody has time now to implement that 15:28:39 <dmcbride> trozet: and bryan_att can we disable those tests on Apex if we do not intent to deliver that functionality? 15:28:44 <trozet> bryan_att: i think it is fine to include it in non-ha, and just document the limitation. It's good to have it any capacity rather than not have it 15:29:08 <bryan_att> I am not suggesting to pull it entirely - only for HA scenarios 15:29:11 <trozet> dmcbride: it's an option if we continue to see the test fail and determine it is not Apex (we have already disabled HA for it) 15:29:20 <trozet> right bryan_att 15:29:20 <narindergupta> dmcbride, i have to goto another meeting. 15:29:25 <bryan_att> dmcbride: the tests are already disabled 15:29:30 <dmcbride> narindergupta: any critical issues to report for joid? 15:30:09 <narindergupta> dmcbride, no last week CI was down but its up now 15:30:20 <narindergupta> and colorado stable releases are passing 15:30:32 <morgan_orange> narinder lxd ha and noha shall turn green as well (at least no more Rally errors) 15:30:32 <bryan_att> btw, there is one potential side-effect of the HA limitation for Congress - it may affect Doctor tests in HA since they are using it I think 15:30:58 <narindergupta> morgan_orange, yes it is 15:31:12 <narindergupta> morgan_orange, only problem is yardstick now 15:31:22 <dmcbride> #info narindergupta reports that colorado stable releases are passing on joid 15:31:49 <dmcbride> someone asked why I was only looking at functest 15:32:18 <dmcbride> I wanted to also look at yardstick, but there is apparently no colorado-specific dashboard for yardstick 15:33:13 <dmcbride> ok - looks like we're out of time 15:33:19 <morgan_orange> http://testresults.opnfv.org/reporting/yardstick/release/stable/colorado/index-status-apex.html 15:33:29 <dmcbride> we will resume same time, same channel on Friday 15:34:18 <dmcbride> morgan_orange: thanks 15:34:28 <dmcbride> morgan_orange: image links appear to be broken 15:34:40 <morgan_orange> dmcbride: they have an issue with the images due to the path in their template 15:34:45 <dmcbride> not seeing the red and green balls 15:34:59 <jose_lausuch> need to drop out sorry 15:35:09 <morgan_orange> they use the same tempalte but some results are in master and the seond one in stable/colorado 15:35:12 <dmcbride> jose_lausuch: thanks - we're done for today 15:35:22 <morgan_orange> I reported it to yardstick 15:35:29 <morgan_orange> but you can see the score.. 15:35:54 <morgan_orange> I can fix it temprarily but tomorrow it will be overwritten if they do not fix the path issue in the template 15:36:02 <dmcbride> morgan_orange: but I like the red and green balls - they're perty ;) 15:36:24 <morgan_orange> but it can be misleading...red with a high score is almost green... 15:36:41 <dmcbride> morgan_orange: yes - I get it 15:37:34 <dmcbride> morgan_orange: looks like much fewer iterations for yardstick as compared to functest 15:38:15 <dmcbride> #endmeeting