13:56:59 <mbeierl> #startmeeting Test and Performance Weekly Meeting
13:56:59 <collabot> Meeting started Thu Aug 25 13:56:59 2016 UTC.  The chair is mbeierl. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:56:59 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
13:56:59 <collabot> The meeting name has been set to 'test_and_performance_weekly_meeting'
13:57:28 <mbeierl> I know, I'm starting a little early, but I have a sudden conflict and would like to get things under way before I have to split my time :)
13:57:37 <mbeierl> #topic Roll Call
13:57:47 <mbeierl> #info Mark Beierl (StorPerf)
13:58:17 <yujunz> #info Yujun Zhang
13:58:22 <yujunz> still in another meeting
13:58:30 <mbeierl> Thanks :)
13:58:55 <morgan_orange> #info Morgan Richomme
13:59:08 <morgan_orange> I initiated the GTM
13:59:25 <mbeierl> I though I started it too...?
13:59:31 <morgan_orange> so I just joined..
14:00:45 <dfarrell07> #info Daniel Farrell (CPerf)
14:00:50 <dfarrell07> (still joining audio)
14:01:51 <morgan_orange> #info agenda https://wiki.opnfv.org/display/meetings/TestPerf
14:02:13 <mbeierl> I'd like to move the discussion of no-ha scenarios towards the start of the meeting if that's ok
14:02:21 <morgan_orange> ok
14:02:50 <yuyang> #info yuyang
14:05:16 <morgan_orange> #topic action items follow up
14:05:25 <morgan_orange> #link http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2016/opnfv-testperf.2016-08-18-13.59.html
14:05:34 <yujunz> It seems I lost audio
14:05:35 <morgan_orange> #info AP1 done
14:05:47 <morgan_orange> #info AP2 mail sent to Frank
14:05:51 <kubi001> yujunz:  same with me
14:06:10 <morgan_orange> #info AP3 pending but we will have more time after C release
14:06:19 <mbeierl> Looks like I lost audio too
14:06:19 <morgan_orange> #action initiate a wiki page on POD allocation for test projects (short/mid/long term) after the C release
14:06:26 * dfarrell07 also hears no one
14:06:38 <mbeierl> Shall we continue IRC only then?
14:06:50 <dfarrell07> IRC is good with me
14:07:01 <kubi001> +1
14:07:12 <yujunz> +1
14:07:15 <acm> #info Al Morton
14:07:21 <kubi001> #info kubi
14:07:35 <dmcbride> #info David McBride
14:07:43 <mbeierl> #info GTM audio issues - meeting moved to IRC only
14:07:50 <acm> I can hear someone typing...
14:07:55 <dfarrell07> I hear background from morgan_orange
14:08:01 <dfarrell07> gone
14:08:06 <dfarrell07> back
14:08:07 <dfarrell07> lol
14:08:10 <kubi001> yes
14:08:57 <acm> morgan and I can hear eachother, perhaps if oyhers re-join???
14:09:10 <kubi001> I can hear you now
14:09:12 <dmcbride> I can hear also
14:09:12 * dfarrell07 rejoins
14:09:33 <morgan_orange> #topic Discussion: should no-ha scenarios be allowed to pass the deploy step by using a virtual environment instead of bare metal?
14:09:56 <acm> audio problems seem to be reconciling now...
14:10:49 <yujunz> I'm not convenient to talk over GTM
14:10:56 <yujunz> But available in IRC
14:11:15 <yujunz> Sorry for that
14:12:54 <dfarrell07> It seems okay to me
14:13:05 <kubi001> ok to me
14:13:12 <yuyang> OK
14:13:16 <morgan_orange> #info on functest we already consider no-ha on virtual environment - e.g. os-odl_l2-moon on virtual env
14:13:45 <yujunz> ok
14:13:50 <dmcbride> are we talking about all testing, or just the deploy portion?
14:14:01 <dfarrell07> #info some consensus that it's okay from dfarrell07, kubi001, yuyang, morgan_orange
14:14:42 <dmcbride> ok - could the chair type in a pound-agreed?
14:15:06 <dfarrell07> dmcbride: I don't think you need to be chair
14:15:09 <morgan_orange> note even some ha scenario are run on virtual env
14:15:20 <kubi001> moon only deploy on virtual env,  so yardstick also test it on virtual env.
14:15:32 <morgan_orange> same for functest
14:15:38 <morgan_orange> deploy and tests are linked
14:15:44 <morgan_orange> we test on what we have
14:16:38 <dmcbride> #info morgan_orange and kubi001 note that some projects are already conducting testing in virtual-only environments (e.g. Moon)
14:16:38 <kubi001> yes, testing depends on the env which the scenario is deployed.
14:17:28 <morgan_orange> for generic scenario (nosdn, odl_l2, onos, ..) it is better to test on target BM env
14:18:10 <dmcbride> #info morgan_orange says:  for generic scenario (nosdn, odl_l2, onos, ..) it is better to test on target BM env
14:18:26 <dmcbride> any dissenting comments?
14:18:34 <dmcbride> any more discussion?
14:18:40 <dfarrell07> none from me
14:18:44 <mbeierl> Sounds good to me :)
14:19:06 <morgan_orange> #info morgan_orange for specific scenario it is acceptable to test on virtual env, it can be indicated in the release note
14:19:55 <morgan_orange> dmcbride: you want an official vote?
14:20:04 <dmcbride> #agreed no-ha scenarios may be tested in a virtual environment.  Test conditions should be identified in the release notes.
14:20:25 <dmcbride> morgan_orange: I don't think that's necessary, unless someone else thinks so
14:20:51 <kubi001> I think only fuel supports noha scenario deployment with jenkins at now.
14:20:57 <dmcbride> is everyone ok with the pound-agreed statement?
14:21:03 <yujunz> ok
14:21:05 <mbeierl> yes
14:21:09 <dfarrell07> yes
14:21:13 <morgan_orange> ok
14:21:20 <acm> ok
14:21:32 <morgan_orange> kubi001:  you have also noha scenarios with joid
14:21:41 <kubi001> morgan_orange: ok
14:21:51 <dmcbride> ok - I think we can move to the next topic
14:22:10 <dmcbride> thanks for your input
14:22:10 <mbeierl> and that would be overall Colorado status
14:22:26 <morgan_orange> #topic overall Colorado status
14:22:36 <mbeierl> thanks, morgan_orange, beat me to it again :)
14:22:53 <mbeierl> I think dmcbride said that everyone's branches have been created
14:23:05 <mbeierl> is there any test project that did not branch?
14:23:21 <dmcbride> mbeierl: yes, that's correct
14:23:38 <mbeierl> ok, StorPerf update
14:23:57 <mbeierl> The intern project is wrapping up this week, with the branch created as of Monday
14:24:21 <mbeierl> this means that what is stable from my intern is in Colorado 1.0, but the final integration is not
14:24:24 <morgan_orange> #info all the test projects branched
14:24:46 <morgan_orange> #info StorPerf update: intern project is wrapping up this week, with the branch created as of Monday
14:25:00 <mbeierl> question: am I allowed to bring forward the final piece, or should I leave that for D release then?
14:25:21 <mbeierl> #info A general question: what exactly does stable mean?  It should be only bug fixes, correct?
14:25:40 <morgan_orange> who can prevent you from bringing a final piece...
14:25:42 <dfarrell07> #info CPerf doesn't have a C branch, but it's intentional (working from master, not in C-release, previously discussed)
14:25:48 <dmcbride> mbeierl: yes, only bug fixes
14:26:12 <mbeierl> morgan_orange: as a PTL, I can prevent me, in the spirit of what dmcbride and the release process states
14:26:22 <mbeierl> I realize there is no formal gate
14:26:50 <yujunz> #info dmcbride confirms only bug fixes are allowed on stable branch
14:27:02 <mbeierl> and that technically, projects can bring forward additional work if they so choose
14:27:13 <mbeierl> but that is not in the spirit of the stable release
14:27:24 <dmcbride> #info only fixes for *documented* bugs
14:27:53 <dfarrell07> I guess you could do it in an SR, since it'd not be a regression or API change?
14:28:38 <mbeierl> That is correct, it is not an API change, but I cannot confirm regression.  The final integration piece would terminate a run when it determines that the values are statistically valid
14:28:47 <mbeierl> and that could be considered a regression change
14:28:55 <mbeierl> or a change in existing behaviour
14:29:33 <dfarrell07> So I guess if it's not a big pain, best for D release
14:30:29 <mbeierl> I think I already decided that, but I also want as a group to help define what our norms are going to be.  We say, *bug* fixes only, but I'd like to agree as a team what that really means
14:31:02 <mbeierl> Does missing functionality become a bug against the current release, for example?
14:31:43 <mbeierl> or do we want to leave that to the PTLs to decide?
14:31:46 <dfarrell07> +1 to guidlines, but I also I think having some flexibility here is valuable
14:31:55 <dfarrell07> Good to talk about it, like this
14:31:56 <dmcbride> +1
14:32:14 <dmcbride> we could discuss this for hours (days)
14:32:25 <mbeierl> I know.  And each case is different
14:32:29 <dmcbride> we need to rely on good judgment of PTLs
14:32:54 <dfarrell07> There are cases when it's really important for some reason to get something into a release, when maybe doing more than a bugfix is very desirable to all
14:33:16 <dfarrell07> Overall, guide to bugfixes, but trust PTLs and feel okay talking about it here
14:33:17 <dmcbride> I would say that new functionality can be considered a bug if the existing functionality is broken without it
14:33:35 <morgan_orange> in a possible future, for me a release will be a snapshot of the existing succesful scenarios.....we even could decide whenever we a=want to declare a release
14:33:49 <dfarrell07> dmcbride: yeah, that's the classic slippery slope here, "needs this feature to work, so kinda a bugfix"
14:33:52 <mbeierl> good point, morgan_orange
14:34:10 <morgan_orange> as we do not have the full maturity to do so, we are following traditionnal software milestones, but we are doing integration/tests not producing a software in a traditionnal way
14:34:25 <mbeierl> the snapshot approach is what agile processes seek to help define
14:34:28 <acm> +1 for PTL judgement, but I also support dmcbride 's point on documentation of "bug"
14:34:36 <dmcbride> dfarrell07: absolutely, so we rely on judgment and good faith of PTLs and project teams to do the right thing
14:35:16 <yujunz> A complete release includes success scenario and known issues, in my opinion.
14:35:26 <dmcbride> yes, we should not be seeing "mystery" commits
14:35:28 <dfarrell07> morgan_orange: big +1 to direction of more CI/CD/CR than normal release, this is kinda what CPerf is getting at working from master
14:35:54 <dmcbride> commits that change code should be associated with a Jira ticket
14:36:49 <mbeierl> dmcbride: that is an excellent point and helps with back tracing.  There should be no cherry picks to stable without a JIRA reference
14:36:51 <mbeierl> oh
14:37:14 <mbeierl> but that brings up my favourite topic: Is there a second JIRA opened for the cherry pick?
14:37:30 <mbeierl> I always was of the opinion: one JIRA, one commit (review).
14:37:53 <morgan_orange> but as for us master = colorado .... one JIRA 2 commits
14:38:07 <mbeierl> That way if it breaks on cherry pick to new branch, the old JIRA is still validly "fixed", but the new one can be re-opened and worked on
14:38:19 <dmcbride> yes, we have enough challenges with JIRA as it is, without introducing more tickets
14:38:40 <mbeierl> morgan_orange: which is why I said "I *was* of that opinion"  I actually see the value of what you say now!
14:38:56 <morgan_orange> if for any reason the cherry pick breaks something...then you can create a new JIRA...but we assume it should not be 'hopeffuly) frequent
14:39:06 <mbeierl> +1
14:39:09 <dfarrell07> +1 to all of that^^
14:39:25 <dmcbride> the only time I think that would be necessary is if there was a substantial divergence between master and stable in the code associated with the ticket
14:39:38 <morgan_orange> keep it simple and push for release tag in the future...branch could be used for dev to work on new features
14:39:46 <mbeierl> and again, we rely of the PTLs to review cherry picks to ensure there is a JIRA
14:40:23 <morgan_orange> #info Functest, branched on the 22nd, troubleshooting in progress on different scenarios with different feature projects. Problems identified in JIRA (jenkins stability in multisite, tempest issues on different scenarios, healthcheck issue with joid/lxd scenarios due to difference in lxc cirrios image,...)
14:40:24 <acm> done and done
14:40:42 <morgan_orange> regarding functest scoring, I have a question
14:41:04 <morgan_orange> according to the success criteria we consider several tiers healthcheck, smoke, sdn and feature
14:41:16 <mbeierl> (which might help lead us to the next agenda topic after: overall Colorado test reporting overview)
14:42:11 <morgan_orange> some fuels scenarios were green, then a feature project added fuel as supported installer but as the tests failes all the scenraio turned red..
14:42:34 <morgan_orange> mbeierl: maybe wait for bottlenecks, yardstick, vsperf update before :)
14:42:46 <mbeierl> morgan_orange: of course
14:43:08 <mbeierl> I did mean to indicate "after we are done with this" :)
14:43:38 <kubi001> Yardstick , branched on this Monday,  fixed all the bugs from yardstick side which we have known,Yardstick created a wiki page to trace the Yardstick ci job status of stable branch.  https://wiki.opnfv.org/display/yardstick/Yardstick+Colorado+CI+Status. Now we are work together with onos team to help them to do some debug work.
14:45:23 <morgan_orange> #info Yardstick , branched on this Monday,  fixed all the bugs from yardstick side which we have known,Yardstick created a wiki page to trace the Yardstick ci job status of stable branch.  https://wiki.opnfv.org/display/yardstick/Yardstick+Colorado+CI+Status. Now we are work together with onos team to help them to do some debug work.
14:45:42 <morgan_orange> yujunz: acm any info on your side?
14:45:55 <mbeierl> kubi001: nice table :)
14:46:08 <kubi001> mbeierl:  :)
14:46:13 <mbeierl> morgan_orange: back to your question, that is interesting as far as schedules go.  Wasn't there supposed to be a milestone cutoff for the situation you describe?
14:46:25 <mbeierl> kubi001: very clear
14:46:32 <yujunz> #info qtip skipped release C and is preparing for release D
14:48:08 <acm> #info  vsperf has added some new tests in C, and support for two new traffic generator/test systems
14:48:51 <morgan_orange> mbeierl:  for feature in generic scenario, there was no milestones saying it will not be there ...the feature has been developped, tests have been provided, doc created...then it is time for integration...so I think it is clear / milestones
14:49:19 <yuyang> #info bottlenecks has updated the docs, fixed the bugs from its side and already stable branched
14:50:25 <morgan_orange> kubi001: nice tables, our experience in brahmaputra is that it was very hard to maintain...you will tell us. that is why we created the automatic reporting for C, we did not want to deal with wiki for reporting.. :)
14:50:52 <mbeierl> Is there a dashboard of results still?
14:51:14 <mbeierl> morgan_orange: I know functest has the high level dashboard with simple pass/fail now
14:51:19 <yujunz> +1 for automatic reporting
14:51:35 <kubi001> morgan_orange: yes, we are developing the reporting function as functest have.
14:51:48 <morgan_orange> #topic Colorado test reporting overview - can someone give a highlight of what jobs and dashboards are available?
14:52:06 <morgan_orange> #info in Functest we distinguish dashboarding and automatic reporting
14:52:20 <kubi001> wiki page is the workaround solution before the reporting html implemented for yardstick
14:52:29 <morgan_orange> #info automatic reporting = http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html
14:53:04 <morgan_orange> #info based on test results stored in the DB and the criteria field (PASS/FAILED) we build automatically these pages to give a consistent view directly from CI/DB
14:53:28 <morgan_orange> #info dashboarding is more test =f (time), we decided for C release to move from home based solution to ELK framework
14:53:39 <morgan_orange> #info we are able to export mongo results into elasticsearch
14:54:01 <morgan_orange> #info we had some issues with elastic search addons (under license) => fixed now
14:54:07 <mbeierl> morgan_orange: looking at the functest dashboard once again makes me think that I should put forward a simple "Cinder Ping" test using StorPerf...
14:54:21 * mbeierl thinks that sounds like a good intern project :)
14:54:49 <morgan_orange> yardstick has also nice dashboards done with grafana
14:55:02 <morgan_orange> so sotrperf surely herited grafana dashboarding
14:55:15 <kubi001> morgan_orange:  automatic reporting is a cool solution for release :)
14:55:37 <morgan_orange> if you push your results to the DB and indicate the overall status of your test, you can easilty use the same automatic reporting
14:55:42 <morgan_orange> code is here
14:55:51 <mbeierl> yes, but we would need to export raw data from StorPerf internal Carbon DB into Yardstick's DN to take advantage of Graphana
14:55:59 <mbeierl> Grafana
14:56:03 <morgan_orange> https://git.opnfv.org/cgit/releng/tree/utils/test/reporting/functest
14:56:35 <kubi001> we have push the result to the DB, now we just to do a little change about the html.
14:57:03 <morgan_orange> it is based on Jinja2
14:57:44 <morgan_orange> thfor reporting it is convenient (as we have a success criteria in the DB), for dashboarding => either ELK or Grafana
14:58:46 <kubi001> morgan_orange: yes, I have read the code  I mean we may do a little change about the test criteria
14:58:47 <morgan_orange> an update on ELK has to be planned and shared with the working group
14:59:01 <morgan_orange> if needed I can also take some minutes on reporting next week if you want
14:59:44 <kubi001> morgan_orange: that's great
14:59:50 <mbeierl> So, for C release, are the two "dashboards" for release status as follows:
14:59:50 <mbeierl> #link https://wiki.opnfv.org/display/yardstick/Yardstick+Colorado+CI+Status
14:59:50 <mbeierl> #link http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html
15:00:31 <morgan_orange> #link http://testresults.opnfv.org/reporting/index-colorado.html
15:00:37 <morgan_orange> if you wnat the link to all the installers
15:01:03 <mbeierl> morgan_orange: oops, thanks.  I accidentally used the drill down :(
15:01:12 <morgan_orange> we can see that we branched recently, there are not lots of iteraion hence a low scoring
15:01:40 <morgan_orange> but sun should come soon...
15:01:59 * mbeierl starts humming "Here comes the sun...."
15:02:13 <morgan_orange> #action morgan_orange prepare short presentation on automatic reporting
15:02:32 <morgan_orange> #action kubi001 morgan_orange plan an update on grafana/ELK in September
15:02:51 <morgan_orange> #topic AoB
15:03:19 <mbeierl> morgan_orange: fyi, clicking on Home in the functest reporting dashboard gives 404: http://testresults.opnfv.org/reporting/functest/release/colorado/index.html
15:03:25 <morgan_orange> #info we have no chair for next week
15:03:48 <kubi001> I'd like to do it .
15:03:54 <mbeierl> I am in Abu Dhabi for the next two weeks, so probably will not be able to participate heavily
15:04:04 <morgan_orange> ok thanks kubi001, I update the wiki
15:04:09 <yujunz> The APAC slot should be on **2nd** Wednesday.
15:04:09 <morgan_orange> I will take the note for you
15:04:17 <kubi001> morgan_orange: thanks
15:04:17 <yujunz> I've corrected the wiki page
15:04:34 <mbeierl> and thanks, morgan_orange for helping me so much today!!
15:05:20 <morgan_orange> yujunz: OK so you may want to chair this session
15:05:48 <morgan_orange> if so do not hesitate to switch between the 14 and the 22
15:05:57 <morgan_orange> any other topic to raise?
15:06:11 <yujunz> Thanks morgan_orange
15:06:19 <morgan_orange> enjoy cherry picking and see you next week :)
15:06:28 <yujunz> See you, bye
15:06:35 <mbeierl> Did we want to draft guidelines about the cherry picking, or just have the discussion here recorded?
15:06:51 <mbeierl> oh, we are over time.  I didn't realize that.
15:06:54 <mbeierl> bye everyone :)
15:06:58 <kubi001> bye
15:07:03 <morgan_orange> you can action yourself, nights are short in Abu Dhabi...
15:07:11 <mbeierl> morgan_orange: true...
15:07:14 <kubi001> :)
15:07:19 <morgan_orange> but life is even shorter..
15:07:45 <mbeierl> #action mbeierl to summarize stable branch guidelines for testperf members
15:07:59 <mbeierl> endmeeting in 30s unless aob?
15:08:03 <yuyang> bye
15:08:27 <mbeierl> 15s
15:08:33 <kubi001> 10s
15:08:36 <mbeierl> 5s
15:08:44 <mbeierl> #endmeeting