13:56:59 <mbeierl> #startmeeting Test and Performance Weekly Meeting 13:56:59 <collabot> Meeting started Thu Aug 25 13:56:59 2016 UTC. The chair is mbeierl. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:56:59 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic. 13:56:59 <collabot> The meeting name has been set to 'test_and_performance_weekly_meeting' 13:57:28 <mbeierl> I know, I'm starting a little early, but I have a sudden conflict and would like to get things under way before I have to split my time :) 13:57:37 <mbeierl> #topic Roll Call 13:57:47 <mbeierl> #info Mark Beierl (StorPerf) 13:58:17 <yujunz> #info Yujun Zhang 13:58:22 <yujunz> still in another meeting 13:58:30 <mbeierl> Thanks :) 13:58:55 <morgan_orange> #info Morgan Richomme 13:59:08 <morgan_orange> I initiated the GTM 13:59:25 <mbeierl> I though I started it too...? 13:59:31 <morgan_orange> so I just joined.. 14:00:45 <dfarrell07> #info Daniel Farrell (CPerf) 14:00:50 <dfarrell07> (still joining audio) 14:01:51 <morgan_orange> #info agenda https://wiki.opnfv.org/display/meetings/TestPerf 14:02:13 <mbeierl> I'd like to move the discussion of no-ha scenarios towards the start of the meeting if that's ok 14:02:21 <morgan_orange> ok 14:02:50 <yuyang> #info yuyang 14:05:16 <morgan_orange> #topic action items follow up 14:05:25 <morgan_orange> #link http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2016/opnfv-testperf.2016-08-18-13.59.html 14:05:34 <yujunz> It seems I lost audio 14:05:35 <morgan_orange> #info AP1 done 14:05:47 <morgan_orange> #info AP2 mail sent to Frank 14:05:51 <kubi001> yujunz: same with me 14:06:10 <morgan_orange> #info AP3 pending but we will have more time after C release 14:06:19 <mbeierl> Looks like I lost audio too 14:06:19 <morgan_orange> #action initiate a wiki page on POD allocation for test projects (short/mid/long term) after the C release 14:06:26 * dfarrell07 also hears no one 14:06:38 <mbeierl> Shall we continue IRC only then? 14:06:50 <dfarrell07> IRC is good with me 14:07:01 <kubi001> +1 14:07:12 <yujunz> +1 14:07:15 <acm> #info Al Morton 14:07:21 <kubi001> #info kubi 14:07:35 <dmcbride> #info David McBride 14:07:43 <mbeierl> #info GTM audio issues - meeting moved to IRC only 14:07:50 <acm> I can hear someone typing... 14:07:55 <dfarrell07> I hear background from morgan_orange 14:08:01 <dfarrell07> gone 14:08:06 <dfarrell07> back 14:08:07 <dfarrell07> lol 14:08:10 <kubi001> yes 14:08:57 <acm> morgan and I can hear eachother, perhaps if oyhers re-join??? 14:09:10 <kubi001> I can hear you now 14:09:12 <dmcbride> I can hear also 14:09:12 * dfarrell07 rejoins 14:09:33 <morgan_orange> #topic Discussion: should no-ha scenarios be allowed to pass the deploy step by using a virtual environment instead of bare metal? 14:09:56 <acm> audio problems seem to be reconciling now... 14:10:49 <yujunz> I'm not convenient to talk over GTM 14:10:56 <yujunz> But available in IRC 14:11:15 <yujunz> Sorry for that 14:12:54 <dfarrell07> It seems okay to me 14:13:05 <kubi001> ok to me 14:13:12 <yuyang> OK 14:13:16 <morgan_orange> #info on functest we already consider no-ha on virtual environment - e.g. os-odl_l2-moon on virtual env 14:13:45 <yujunz> ok 14:13:50 <dmcbride> are we talking about all testing, or just the deploy portion? 14:14:01 <dfarrell07> #info some consensus that it's okay from dfarrell07, kubi001, yuyang, morgan_orange 14:14:42 <dmcbride> ok - could the chair type in a pound-agreed? 14:15:06 <dfarrell07> dmcbride: I don't think you need to be chair 14:15:09 <morgan_orange> note even some ha scenario are run on virtual env 14:15:20 <kubi001> moon only deploy on virtual env, so yardstick also test it on virtual env. 14:15:32 <morgan_orange> same for functest 14:15:38 <morgan_orange> deploy and tests are linked 14:15:44 <morgan_orange> we test on what we have 14:16:38 <dmcbride> #info morgan_orange and kubi001 note that some projects are already conducting testing in virtual-only environments (e.g. Moon) 14:16:38 <kubi001> yes, testing depends on the env which the scenario is deployed. 14:17:28 <morgan_orange> for generic scenario (nosdn, odl_l2, onos, ..) it is better to test on target BM env 14:18:10 <dmcbride> #info morgan_orange says: for generic scenario (nosdn, odl_l2, onos, ..) it is better to test on target BM env 14:18:26 <dmcbride> any dissenting comments? 14:18:34 <dmcbride> any more discussion? 14:18:40 <dfarrell07> none from me 14:18:44 <mbeierl> Sounds good to me :) 14:19:06 <morgan_orange> #info morgan_orange for specific scenario it is acceptable to test on virtual env, it can be indicated in the release note 14:19:55 <morgan_orange> dmcbride: you want an official vote? 14:20:04 <dmcbride> #agreed no-ha scenarios may be tested in a virtual environment. Test conditions should be identified in the release notes. 14:20:25 <dmcbride> morgan_orange: I don't think that's necessary, unless someone else thinks so 14:20:51 <kubi001> I think only fuel supports noha scenario deployment with jenkins at now. 14:20:57 <dmcbride> is everyone ok with the pound-agreed statement? 14:21:03 <yujunz> ok 14:21:05 <mbeierl> yes 14:21:09 <dfarrell07> yes 14:21:13 <morgan_orange> ok 14:21:20 <acm> ok 14:21:32 <morgan_orange> kubi001: you have also noha scenarios with joid 14:21:41 <kubi001> morgan_orange: ok 14:21:51 <dmcbride> ok - I think we can move to the next topic 14:22:10 <dmcbride> thanks for your input 14:22:10 <mbeierl> and that would be overall Colorado status 14:22:26 <morgan_orange> #topic overall Colorado status 14:22:36 <mbeierl> thanks, morgan_orange, beat me to it again :) 14:22:53 <mbeierl> I think dmcbride said that everyone's branches have been created 14:23:05 <mbeierl> is there any test project that did not branch? 14:23:21 <dmcbride> mbeierl: yes, that's correct 14:23:38 <mbeierl> ok, StorPerf update 14:23:57 <mbeierl> The intern project is wrapping up this week, with the branch created as of Monday 14:24:21 <mbeierl> this means that what is stable from my intern is in Colorado 1.0, but the final integration is not 14:24:24 <morgan_orange> #info all the test projects branched 14:24:46 <morgan_orange> #info StorPerf update: intern project is wrapping up this week, with the branch created as of Monday 14:25:00 <mbeierl> question: am I allowed to bring forward the final piece, or should I leave that for D release then? 14:25:21 <mbeierl> #info A general question: what exactly does stable mean? It should be only bug fixes, correct? 14:25:40 <morgan_orange> who can prevent you from bringing a final piece... 14:25:42 <dfarrell07> #info CPerf doesn't have a C branch, but it's intentional (working from master, not in C-release, previously discussed) 14:25:48 <dmcbride> mbeierl: yes, only bug fixes 14:26:12 <mbeierl> morgan_orange: as a PTL, I can prevent me, in the spirit of what dmcbride and the release process states 14:26:22 <mbeierl> I realize there is no formal gate 14:26:50 <yujunz> #info dmcbride confirms only bug fixes are allowed on stable branch 14:27:02 <mbeierl> and that technically, projects can bring forward additional work if they so choose 14:27:13 <mbeierl> but that is not in the spirit of the stable release 14:27:24 <dmcbride> #info only fixes for *documented* bugs 14:27:53 <dfarrell07> I guess you could do it in an SR, since it'd not be a regression or API change? 14:28:38 <mbeierl> That is correct, it is not an API change, but I cannot confirm regression. The final integration piece would terminate a run when it determines that the values are statistically valid 14:28:47 <mbeierl> and that could be considered a regression change 14:28:55 <mbeierl> or a change in existing behaviour 14:29:33 <dfarrell07> So I guess if it's not a big pain, best for D release 14:30:29 <mbeierl> I think I already decided that, but I also want as a group to help define what our norms are going to be. We say, *bug* fixes only, but I'd like to agree as a team what that really means 14:31:02 <mbeierl> Does missing functionality become a bug against the current release, for example? 14:31:43 <mbeierl> or do we want to leave that to the PTLs to decide? 14:31:46 <dfarrell07> +1 to guidlines, but I also I think having some flexibility here is valuable 14:31:55 <dfarrell07> Good to talk about it, like this 14:31:56 <dmcbride> +1 14:32:14 <dmcbride> we could discuss this for hours (days) 14:32:25 <mbeierl> I know. And each case is different 14:32:29 <dmcbride> we need to rely on good judgment of PTLs 14:32:54 <dfarrell07> There are cases when it's really important for some reason to get something into a release, when maybe doing more than a bugfix is very desirable to all 14:33:16 <dfarrell07> Overall, guide to bugfixes, but trust PTLs and feel okay talking about it here 14:33:17 <dmcbride> I would say that new functionality can be considered a bug if the existing functionality is broken without it 14:33:35 <morgan_orange> in a possible future, for me a release will be a snapshot of the existing succesful scenarios.....we even could decide whenever we a=want to declare a release 14:33:49 <dfarrell07> dmcbride: yeah, that's the classic slippery slope here, "needs this feature to work, so kinda a bugfix" 14:33:52 <mbeierl> good point, morgan_orange 14:34:10 <morgan_orange> as we do not have the full maturity to do so, we are following traditionnal software milestones, but we are doing integration/tests not producing a software in a traditionnal way 14:34:25 <mbeierl> the snapshot approach is what agile processes seek to help define 14:34:28 <acm> +1 for PTL judgement, but I also support dmcbride 's point on documentation of "bug" 14:34:36 <dmcbride> dfarrell07: absolutely, so we rely on judgment and good faith of PTLs and project teams to do the right thing 14:35:16 <yujunz> A complete release includes success scenario and known issues, in my opinion. 14:35:26 <dmcbride> yes, we should not be seeing "mystery" commits 14:35:28 <dfarrell07> morgan_orange: big +1 to direction of more CI/CD/CR than normal release, this is kinda what CPerf is getting at working from master 14:35:54 <dmcbride> commits that change code should be associated with a Jira ticket 14:36:49 <mbeierl> dmcbride: that is an excellent point and helps with back tracing. There should be no cherry picks to stable without a JIRA reference 14:36:51 <mbeierl> oh 14:37:14 <mbeierl> but that brings up my favourite topic: Is there a second JIRA opened for the cherry pick? 14:37:30 <mbeierl> I always was of the opinion: one JIRA, one commit (review). 14:37:53 <morgan_orange> but as for us master = colorado .... one JIRA 2 commits 14:38:07 <mbeierl> That way if it breaks on cherry pick to new branch, the old JIRA is still validly "fixed", but the new one can be re-opened and worked on 14:38:19 <dmcbride> yes, we have enough challenges with JIRA as it is, without introducing more tickets 14:38:40 <mbeierl> morgan_orange: which is why I said "I *was* of that opinion" I actually see the value of what you say now! 14:38:56 <morgan_orange> if for any reason the cherry pick breaks something...then you can create a new JIRA...but we assume it should not be 'hopeffuly) frequent 14:39:06 <mbeierl> +1 14:39:09 <dfarrell07> +1 to all of that^^ 14:39:25 <dmcbride> the only time I think that would be necessary is if there was a substantial divergence between master and stable in the code associated with the ticket 14:39:38 <morgan_orange> keep it simple and push for release tag in the future...branch could be used for dev to work on new features 14:39:46 <mbeierl> and again, we rely of the PTLs to review cherry picks to ensure there is a JIRA 14:40:23 <morgan_orange> #info Functest, branched on the 22nd, troubleshooting in progress on different scenarios with different feature projects. Problems identified in JIRA (jenkins stability in multisite, tempest issues on different scenarios, healthcheck issue with joid/lxd scenarios due to difference in lxc cirrios image,...) 14:40:24 <acm> done and done 14:40:42 <morgan_orange> regarding functest scoring, I have a question 14:41:04 <morgan_orange> according to the success criteria we consider several tiers healthcheck, smoke, sdn and feature 14:41:16 <mbeierl> (which might help lead us to the next agenda topic after: overall Colorado test reporting overview) 14:42:11 <morgan_orange> some fuels scenarios were green, then a feature project added fuel as supported installer but as the tests failes all the scenraio turned red.. 14:42:34 <morgan_orange> mbeierl: maybe wait for bottlenecks, yardstick, vsperf update before :) 14:42:46 <mbeierl> morgan_orange: of course 14:43:08 <mbeierl> I did mean to indicate "after we are done with this" :) 14:43:38 <kubi001> Yardstick , branched on this Monday, fixed all the bugs from yardstick side which we have known,Yardstick created a wiki page to trace the Yardstick ci job status of stable branch. https://wiki.opnfv.org/display/yardstick/Yardstick+Colorado+CI+Status. Now we are work together with onos team to help them to do some debug work. 14:45:23 <morgan_orange> #info Yardstick , branched on this Monday, fixed all the bugs from yardstick side which we have known,Yardstick created a wiki page to trace the Yardstick ci job status of stable branch. https://wiki.opnfv.org/display/yardstick/Yardstick+Colorado+CI+Status. Now we are work together with onos team to help them to do some debug work. 14:45:42 <morgan_orange> yujunz: acm any info on your side? 14:45:55 <mbeierl> kubi001: nice table :) 14:46:08 <kubi001> mbeierl: :) 14:46:13 <mbeierl> morgan_orange: back to your question, that is interesting as far as schedules go. Wasn't there supposed to be a milestone cutoff for the situation you describe? 14:46:25 <mbeierl> kubi001: very clear 14:46:32 <yujunz> #info qtip skipped release C and is preparing for release D 14:48:08 <acm> #info vsperf has added some new tests in C, and support for two new traffic generator/test systems 14:48:51 <morgan_orange> mbeierl: for feature in generic scenario, there was no milestones saying it will not be there ...the feature has been developped, tests have been provided, doc created...then it is time for integration...so I think it is clear / milestones 14:49:19 <yuyang> #info bottlenecks has updated the docs, fixed the bugs from its side and already stable branched 14:50:25 <morgan_orange> kubi001: nice tables, our experience in brahmaputra is that it was very hard to maintain...you will tell us. that is why we created the automatic reporting for C, we did not want to deal with wiki for reporting.. :) 14:50:52 <mbeierl> Is there a dashboard of results still? 14:51:14 <mbeierl> morgan_orange: I know functest has the high level dashboard with simple pass/fail now 14:51:19 <yujunz> +1 for automatic reporting 14:51:35 <kubi001> morgan_orange: yes, we are developing the reporting function as functest have. 14:51:48 <morgan_orange> #topic Colorado test reporting overview - can someone give a highlight of what jobs and dashboards are available? 14:52:06 <morgan_orange> #info in Functest we distinguish dashboarding and automatic reporting 14:52:20 <kubi001> wiki page is the workaround solution before the reporting html implemented for yardstick 14:52:29 <morgan_orange> #info automatic reporting = http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html 14:53:04 <morgan_orange> #info based on test results stored in the DB and the criteria field (PASS/FAILED) we build automatically these pages to give a consistent view directly from CI/DB 14:53:28 <morgan_orange> #info dashboarding is more test =f (time), we decided for C release to move from home based solution to ELK framework 14:53:39 <morgan_orange> #info we are able to export mongo results into elasticsearch 14:54:01 <morgan_orange> #info we had some issues with elastic search addons (under license) => fixed now 14:54:07 <mbeierl> morgan_orange: looking at the functest dashboard once again makes me think that I should put forward a simple "Cinder Ping" test using StorPerf... 14:54:21 * mbeierl thinks that sounds like a good intern project :) 14:54:49 <morgan_orange> yardstick has also nice dashboards done with grafana 14:55:02 <morgan_orange> so sotrperf surely herited grafana dashboarding 14:55:15 <kubi001> morgan_orange: automatic reporting is a cool solution for release :) 14:55:37 <morgan_orange> if you push your results to the DB and indicate the overall status of your test, you can easilty use the same automatic reporting 14:55:42 <morgan_orange> code is here 14:55:51 <mbeierl> yes, but we would need to export raw data from StorPerf internal Carbon DB into Yardstick's DN to take advantage of Graphana 14:55:59 <mbeierl> Grafana 14:56:03 <morgan_orange> https://git.opnfv.org/cgit/releng/tree/utils/test/reporting/functest 14:56:35 <kubi001> we have push the result to the DB, now we just to do a little change about the html. 14:57:03 <morgan_orange> it is based on Jinja2 14:57:44 <morgan_orange> thfor reporting it is convenient (as we have a success criteria in the DB), for dashboarding => either ELK or Grafana 14:58:46 <kubi001> morgan_orange: yes, I have read the code I mean we may do a little change about the test criteria 14:58:47 <morgan_orange> an update on ELK has to be planned and shared with the working group 14:59:01 <morgan_orange> if needed I can also take some minutes on reporting next week if you want 14:59:44 <kubi001> morgan_orange: that's great 14:59:50 <mbeierl> So, for C release, are the two "dashboards" for release status as follows: 14:59:50 <mbeierl> #link https://wiki.opnfv.org/display/yardstick/Yardstick+Colorado+CI+Status 14:59:50 <mbeierl> #link http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html 15:00:31 <morgan_orange> #link http://testresults.opnfv.org/reporting/index-colorado.html 15:00:37 <morgan_orange> if you wnat the link to all the installers 15:01:03 <mbeierl> morgan_orange: oops, thanks. I accidentally used the drill down :( 15:01:12 <morgan_orange> we can see that we branched recently, there are not lots of iteraion hence a low scoring 15:01:40 <morgan_orange> but sun should come soon... 15:01:59 * mbeierl starts humming "Here comes the sun...." 15:02:13 <morgan_orange> #action morgan_orange prepare short presentation on automatic reporting 15:02:32 <morgan_orange> #action kubi001 morgan_orange plan an update on grafana/ELK in September 15:02:51 <morgan_orange> #topic AoB 15:03:19 <mbeierl> morgan_orange: fyi, clicking on Home in the functest reporting dashboard gives 404: http://testresults.opnfv.org/reporting/functest/release/colorado/index.html 15:03:25 <morgan_orange> #info we have no chair for next week 15:03:48 <kubi001> I'd like to do it . 15:03:54 <mbeierl> I am in Abu Dhabi for the next two weeks, so probably will not be able to participate heavily 15:04:04 <morgan_orange> ok thanks kubi001, I update the wiki 15:04:09 <yujunz> The APAC slot should be on **2nd** Wednesday. 15:04:09 <morgan_orange> I will take the note for you 15:04:17 <kubi001> morgan_orange: thanks 15:04:17 <yujunz> I've corrected the wiki page 15:04:34 <mbeierl> and thanks, morgan_orange for helping me so much today!! 15:05:20 <morgan_orange> yujunz: OK so you may want to chair this session 15:05:48 <morgan_orange> if so do not hesitate to switch between the 14 and the 22 15:05:57 <morgan_orange> any other topic to raise? 15:06:11 <yujunz> Thanks morgan_orange 15:06:19 <morgan_orange> enjoy cherry picking and see you next week :) 15:06:28 <yujunz> See you, bye 15:06:35 <mbeierl> Did we want to draft guidelines about the cherry picking, or just have the discussion here recorded? 15:06:51 <mbeierl> oh, we are over time. I didn't realize that. 15:06:54 <mbeierl> bye everyone :) 15:06:58 <kubi001> bye 15:07:03 <morgan_orange> you can action yourself, nights are short in Abu Dhabi... 15:07:11 <mbeierl> morgan_orange: true... 15:07:14 <kubi001> :) 15:07:19 <morgan_orange> but life is even shorter.. 15:07:45 <mbeierl> #action mbeierl to summarize stable branch guidelines for testperf members 15:07:59 <mbeierl> endmeeting in 30s unless aob? 15:08:03 <yuyang> bye 15:08:27 <mbeierl> 15s 15:08:33 <kubi001> 10s 15:08:36 <mbeierl> 5s 15:08:44 <mbeierl> #endmeeting