08:00:10 <jose_lausuch> #startmeeting Functest weekly meeting Marc 21th 2017
08:00:10 <collabot`> Meeting started Tue Mar 21 08:00:10 2017 UTC.  The chair is jose_lausuch. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:00:10 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:00:10 <collabot`> The meeting name has been set to 'functest_weekly_meeting_marc_21th_2017'
08:00:10 <collabot`> morgan_orange: Error: Can't start another meeting, one is in progress.  Use #endmeeting first.
08:00:15 <jose_lausuch> how are you?
08:00:18 <jose_lausuch> oops
08:00:20 <morgan_orange> fine and you
08:00:23 <LindaWang> jose_lausuch: welcome back. :)
08:00:29 <jose_lausuch> we started 2 meetings :D
08:00:35 <jose_lausuch> thanks
08:00:48 <jose_lausuch> go ahead morgan
08:00:50 <morgan_orange> but the bot is smart, it just created one
08:01:09 <morgan_orange> #topic call rol
08:01:16 <morgan_orange> #info Morgan Richomme
08:01:20 <jose_lausuch> #info Jose Lausuch
08:01:29 <juhak> #info Juha Kosonen
08:01:39 <HelenYao> #info Helen Yao
08:01:47 <morgan_orange> #info agenda https://wiki.opnfv.org/display/functest/Functest+Meeting
08:02:03 <morgan_orange> mainly on Danube  and CFP beinjing
08:02:12 <LindaWang> #info Linda Wang
08:02:17 <morgan_orange> #topic action point follow-up
08:02:19 <hideyasu> #inf Hideyasu
08:02:21 <ollivier> #info Cédric Ollivier
08:02:23 <morgan_orange> #info AP1: ollivier add a wiki page about review best practices
08:02:29 <morgan_orange> #action ollivier add a wiki page about review best practices
08:02:33 <morgan_orange> #info AP2: HelenYao generate new functest:stable based on danube
08:02:41 <HelenYao> done
08:02:44 <morgan_orange> #info done patch done by HelenYao + patch to remove ovno (no Danube branche)
08:02:50 <morgan_orange> #info AP3: morgan_orange contact bgpvpn team for apex
08:02:56 <morgan_orange> #info note done because no bgpvpn on Apex until yesterday...
08:03:04 <morgan_orange> #info wait & see next run and see next topic
08:03:09 <morgan_orange> #info morgan_orange prepare release note
08:03:11 <jose_lausuch> is AP2 targetting the dockerfile?
08:03:18 <morgan_orange> #info not done, rework on user and config guide
08:03:49 <morgan_orange> jose_lausuch: yes tehre was a problem after Danube branch
08:03:57 <morgan_orange> tests were using docker: stable
08:04:01 <morgan_orange> which was still Colorado based
08:04:05 <jose_lausuch> ok
08:04:12 <jose_lausuch> thanks
08:04:13 <morgan_orange> so we reenabled he job to produce a stable Danube
08:04:23 <morgan_orange> HelenYao: fixed the jobs
08:04:33 <jose_lausuch> we need to maintain 2 dockerfiles now
08:04:38 <morgan_orange> and we got another issue due to the fact that we were trying to clone ovno
08:04:44 <morgan_orange> but ovno was not declared for Danube
08:04:46 <jose_lausuch> 1 pointing to master, 1 pointint to danube branch
08:04:51 <morgan_orange> opnfvdocs had the same issue
08:04:54 <jose_lausuch> ok
08:04:55 <morgan_orange> jose_lausuch: yes
08:05:11 <morgan_orange> #action morgan_orange jose_lausuch prepare release note
08:05:17 <morgan_orange> #info AP4: morgan_orange initiate a wiki page for Functest@plugfest - plan remote demo for stress tests
08:05:22 <morgan_orange> #info done
08:05:26 <morgan_orange> #link https://wiki.opnfv.org/pages/viewpage.action?pageId=9569132
08:05:35 <morgan_orange> any comment on the action point from last week?
08:05:57 <morgan_orange> #topic Danube
08:06:18 <morgan_orange> #info all the installers have now Danube deployment (since last week end)
08:06:57 <jose_lausuch> cool]
08:07:06 <morgan_orange> #link http://testresults.opnfv.org/reporting/functest/release/danube/index-status-apex.html
08:07:13 <jose_lausuch> 1 week to release... it was time..
08:07:24 <morgan_orange> yes as said to David we reached MS3
08:07:37 <morgan_orange> initially planned beginning of January...
08:07:53 <morgan_orange> fortunately we were able to work on master but it is very short for integration...
08:08:24 <jose_lausuch> next release milestones will be mre strict
08:09:11 <jose_lausuch> I read that Doctor is back to life
08:09:14 <jose_lausuch> at least in fuel
08:09:21 <jose_lausuch> not supported by joid?
08:09:36 <morgan_orange> no disabled on joid
08:10:11 <jose_lausuch> and still failing in apex?
08:10:41 <morgan_orange> not systematically
08:10:48 <morgan_orange> see apex release page, it is OK with some scenarios
08:10:50 <jose_lausuch> ok
08:10:59 <morgan_orange> #info odl scenario versus tempest: it seems that we have systematically error in tempest_smoke_serials
08:11:11 <morgan_orange> #info patch pending to exclude some tests for fuel
08:11:30 <morgan_orange> shall we exclude some tests for all the installers in odl scenarios?
08:11:45 <jose_lausuch> which tests are failing?
08:11:52 <jose_lausuch> nova_manage_servers?
08:11:55 <morgan_orange> https://gerrit.opnfv.org/gerrit/#/c/30341/2
08:11:59 <juhak> it's unclear why there is a problem only and always in fuel/odl
08:11:59 <morgan_orange> #link https://gerrit.opnfv.org/gerrit/#/c/30341/2
08:12:06 <juhak> but not in other installers/scenarios
08:12:12 <morgan_orange> juhak: is it always witj fuel
08:12:17 <juhak> yes
08:12:23 <morgan_orange> I can see also that the case is failed on apex scenario
08:12:39 <jose_lausuch> the famous basic_ops :)
08:12:44 <juhak> is it the same root cause?
08:13:10 <ollivier> It doesn't seem related to ODL
08:13:21 <juhak> morgan_orange: can you share the link to failed apex case?
08:13:25 <morgan_orange> not sure, and just checking except odl_fdio, odl tests seem OK
08:13:34 <morgan_orange> http://testresults.opnfv.org/reporting/functest/release/danube/index-status-apex.html
08:14:12 <morgan_orange> seem they are better now (not at the same weather level than other showing that something has been corrected meanwhile)  but no more storm
08:14:39 <morgan_orange> ok so no systematic error so  let's keep as it is
08:14:44 <jose_lausuch> shouldn't we execute refastack test only on nosdn scenario?
08:15:10 <morgan_orange> #info no tempest systematic error on odl scenario so  let's keep as it is
08:15:23 <ollivier> We should skip (or keep) for every scenario if the test is unclear. It's not directly related to ODL.
08:16:23 <morgan_orange> juhak: on compass we have also not 100% on tempest_smoke_serial https://build.opnfv.org/ci/view/functest/job/functest-compass-baremetal-daily-danube/20/console
08:17:00 <morgan_orange> jose_lausuch: I asked the question
08:17:19 <morgan_orange> there is also a patch to exclude tempest_defcore (duplication)
08:17:31 <morgan_orange> https://gerrit.opnfv.org/gerrit/#/c/30735/
08:17:54 <morgan_orange> as far as I understood, refstack is used by dovetail for certification of scenario => so all scenarios
08:18:05 <morgan_orange> #info refstack suite now integrated
08:18:06 <jose_lausuch> yes, tempest defcore is the former test until refstack was integrated
08:18:18 <morgan_orange> #info criteria set by dovetail > 80% (errors are accepted..)
08:18:22 <jose_lausuch> morgan_orange: ok
08:18:49 <morgan_orange> #action jose_lausuch exclude tempest_defcore (we can keep the code as it can be usefull to have a second launcher in case) but not execute it
08:18:55 <morgan_orange> #link https://gerrit.opnfv.org/gerrit/#/c/30735/
08:19:35 <morgan_orange> I think that refscatck is failed on FD scenario, because they functest is not up to date (Cisco labs not connected to CI) they probably expect 80ù (bug fixed by LindaWang)
08:20:03 <juhak> morgan_orange: some sort of connectivity problems, I'll create ticket to compass team
08:20:03 <jose_lausuch> ok
08:20:05 <morgan_orange> #action LindaWang contact juraj to indicate to update functest docker or at least to change the refstack criteria
08:20:27 <morgan_orange> #info request to support https
08:20:36 <morgan_orange> #info pending patch provided by HelenYao
08:20:51 <morgan_orange> #link https://gerrit.opnfv.org/gerrit/#/c/30253/
08:21:50 <jose_lausuch> can you comment on this issue?
08:21:50 <morgan_orange> any question on refstack?
08:21:55 <jose_lausuch> what happened?
08:21:59 <LindaWang> sure
08:23:22 <morgan_orange> maybe you can do it in the IRC..
08:23:34 <LindaWang> ok , i will try.
08:23:55 <jose_lausuch> LindaWang: we can take it later
08:23:57 <jose_lausuch> let's continue
08:24:50 <morgan_orange> #info JIRA status (filter on JIRA bug or task opened in Danube, supposed to be fixed in Danube) see https://wiki.opnfv.org/display/functest/Functest+Meeting
08:25:03 <morgan_orange> in theory it shall converge to 0 with the release
08:25:40 <morgan_orange> either fixed or postponed to danube 2.0, or 3.0 or Euphrates 5.0 (naming changed 5 = 5th version)
08:26:44 <jose_lausuch> we need tos tart closing all danuve 1.09
08:26:46 <jose_lausuch> 1.0
08:26:52 <jose_lausuch> danube1.0
08:26:58 <jose_lausuch> sorry
08:27:00 <morgan_orange> #action HelenYao ollivier jose_lausuch morgan_orange juhak LindaWang boucherv take care for your danube 1.0 JIRA
08:27:20 <jose_lausuch> we need to
08:27:24 <jose_lausuch> I'll remind you this week
08:27:42 <jose_lausuch> by monday, there shouldn't be anything in that list
08:27:50 <jose_lausuch> at least for danube1.0
08:28:11 <morgan_orange> ok next point is on documentation
08:28:29 <morgan_orange> I sent an new version end of last week, I am currently redoing the figures
08:28:33 <morgan_orange> I will amend today
08:29:00 <morgan_orange> I think documenation shall be finalized for the 24th
08:29:23 <jose_lausuch> ok, I'll review it today/tomorrow
08:29:27 <morgan_orange> for user/config/dev guide it should be OK
08:29:39 <jose_lausuch> what about release notes ? also 24th?
08:29:47 <morgan_orange> next poitn will be the release note assuming that the test env is not stabilized yet (not 4 iterations for all the scenarios)
08:29:47 <jose_lausuch> I guess so, release is on 27th
08:29:58 <jose_lausuch> we have to wait until the last day
08:30:10 <jose_lausuch> to get the results
08:30:18 <morgan_orange> yes but we can start ...
08:30:42 <jose_lausuch> agree
08:30:46 <morgan_orange> we shall also make a snapshort of the reporting on the 27th
08:31:02 <morgan_orange> I am not fully clear yet with bgpvpn, barometer and vnf tests
08:31:27 <morgan_orange> we got a weekly job since 1 week, still some work in progress on cloudify_ims, orchestra_ims, vyos-vrouter
08:31:44 <morgan_orange> hideyasu:  do you know when your patch will be available?
08:31:57 <hideyasu> I have fixed vrouter with an v3 API of OpenStack at 3/20 20:00:(UTC) and it show PASS on my lab.
08:32:17 <hideyasu> The fixed point is ool git.
08:32:22 <morgan_orange> Ericsson POD2 is now heavily used, not easy to find a slot to trigger a weekly job
08:32:41 <hideyasu> I fixed.
08:33:09 <morgan_orange> OK orchestra also provided a patch so I assume next run shall show PASS...
08:33:36 <hideyasu> yes. I hope to pass next Jenkins CI test.
08:33:37 <morgan_orange> #info documentation updated - review on config, installation guide in progress
08:33:57 <morgan_orange> #info release note to be initiated but wait the last day to finalize with last test status
08:34:13 <morgan_orange> #action morgan_orange update the figures of the guides
08:34:19 <morgan_orange> ok thanks hideyasu
08:34:38 <morgan_orange> last point for danube was the request from SFC to collect logs of aborted jenkins jobs
08:34:55 <morgan_orange> #info colelction of logs of aborted jobs: jira open (as blocker...)
08:35:44 <morgan_orange> HelenYao: jose_lausuch your view? I saw the comment from fdegir. It is too late to implement our own timeout mechanism through the abstraction (could be possible in Euphrates)
08:35:53 <jose_lausuch> #action jose_lausuch check on bgpvpn status
08:36:33 <jose_lausuch> morgan_orange: havent read latest emails, but I thought the timeout could be handled easier
08:36:36 <morgan_orange> jose_lausuch: scenario on apex => issue on bgpvpn and tempest_smoke
08:36:38 <HelenYao> jose_lausuch: I will leave my point on JIRA. Need to think about it
08:37:16 <jose_lausuch> morgan_orange: ok,I'll take a look
08:37:58 <jose_lausuch> jose_lausuch: but they are blind because of this...
08:38:00 <morgan_orange> same issues with fuel (tempest_smoke_serial + bgpvpn)
08:38:28 <morgan_orange> and as far as I understood, they cannot reproduce on a community lab
08:38:31 <jose_lausuch> for tempest, it's maybe the basic_ops again
08:39:31 <morgan_orange> yes
08:39:31 <morgan_orange> http://testresults.opnfv.org/test/api/v1/results/58cf5c6832c829000a1150c3
08:40:37 <morgan_orange> and yes http://testresults.opnfv.org/test/api/v1/results/58d01e3e32c829000a1151e4
08:40:48 <morgan_orange> same error also on odl scenario on compass
08:41:02 <jose_lausuch> shall we then disable basic_ops?
08:41:18 <morgan_orange> it is OK on several scenarios...
08:42:03 <morgan_orange> the goal is not to have blue balls...is there is one good reason to exclude the test case, it is possible but having blue balls is not one :)
08:42:30 <jose_lausuch> regarding the timeouts...
08:42:35 <morgan_orange> juhak: do we know why this test case is most of the time the one failing?
08:42:37 <jose_lausuch> since we now execute refstack on all scenarios...
08:42:44 <jose_lausuch> maybe we should increase timeouts...
08:43:12 <juhak> apex issue is in network_basic_ops whilst in fuel/odl server_basic_ops
08:44:39 <morgan_orange> refstack has more tests but is less strict (criteria > 80% versus 100% for tempest_smoke_serial)
08:45:06 <morgan_orange> I prefer to be more strict...at the moment even if there are errors in refstack we may say PASS
08:45:26 <morgan_orange> LindaWang: do you know why they set the criteria to > 80% for refstack?
08:45:58 <morgan_orange> MatthewLi: ?
08:46:11 <LindaWang> Dovetail just want the refstack to be cherry picked to D release.
08:46:30 <MatthewLi> #info Jun Li
08:46:46 <morgan_orange> LindaWang: it is already done...
08:46:47 <MatthewLi> more strict is OK for me
08:47:16 <jose_lausuch> MatthewLi: but why was decided 80% ?
08:47:22 <jose_lausuch> and not 90 or 100% ?
08:47:28 <MatthewLi> just set a number ....
08:47:37 <jose_lausuch> that number is kind of important :)
08:47:38 <morgan_orange> we had this disucssion some time ago on tempest, returning PASS and a Jenkins blue ball when there are errors is misleading for me
08:48:11 <MatthewLi> for dovetail we don't quite care the pass percent, we care which testcase pass and which fails....
08:48:22 <MatthewLi> so any number is OK...
08:48:39 <morgan_orange> you do not have any threshold?
08:48:49 <jose_lausuch> MatthewLi: but that defines if the whole suite pass or not
08:49:01 <jose_lausuch> do you do it manually looking at each test case?
08:49:24 <MatthewLi> jose_lausuch: #link http://testresults.opnfv.org/test/swagger/spec.html#!/spec/queryTestResults
08:49:25 <morgan_orange> if you do not consider the figure, I would suggest to set it to 100% to be consistent with our other tests..
08:49:34 <MatthewLi> usually I use this one to see
08:49:40 <morgan_orange> so 1 failure in refstack >+ red status in CI
08:49:49 <MatthewLi> morgan_orange: ok 100% is OK for me
08:50:13 <morgan_orange> ok, no objection?
08:50:16 <jose_lausuch> MatthewLi: if you set it to 100%, you don't need to ask the API for that then
08:50:21 <jose_lausuch> nope
08:50:30 <morgan_orange> #action LindaWang set Refstack criteria to 100%
08:50:34 <LindaWang> Since that, i will revert the criteria to 100%.
08:50:35 <LindaWang> sure
08:50:54 <morgan_orange> #topic CFP Beijing
08:51:01 <morgan_orange> #info deadline 27th of March
08:51:11 <morgan_orange> #info testing group suggested several proposals
08:51:20 <morgan_orange> #link https://etherpad.opnfv.org/p/TestCommunitySummit
08:51:35 <morgan_orange> what do we do for Functest...potentially several topics
08:52:02 <morgan_orange> I think sharing the work done on abstraction, unit testing makes sense in design summit
08:52:19 <morgan_orange> all the work done also by rohit, ashish or kumar in the internship
08:52:32 <jose_lausuch> yes
08:52:49 <jose_lausuch> do we want to have another generic presentation about functest? :)
08:52:57 <morgan_orange> best practices/errors , X community work ..
08:53:09 <jose_lausuch> many rivers to cross....
08:53:45 <morgan_orange> exactly..but regarding the other projects I think we can share some best practices and also some hard points (logger, configuration,...)
08:53:51 <morgan_orange> it is valuable for all the community
08:54:22 <morgan_orange> if you had to select 1 topic for the summit related to Functest what would it be?
08:54:44 <OPNFV-Gerrit-Bot> valentin boucher proposed functest: [cloudify_ims] Change VNF flavor value  https://gerrit.opnfv.org/gerrit/31147
08:54:48 <jose_lausuch> integration of feature tests
08:54:49 <morgan_orange> HelenYao: LindaWang SerenaFeng ollivier jose_lausuch juhak boucherv hideyasu jose_lausuch
08:55:01 <jose_lausuch> how easier it has become
08:56:03 <HelenYao> yes, the refactor of the framework is a good candidate topic
08:56:37 <morgan_orange> OK it seems that it is a good topic for a design summit, we have also work to do to be more strict on the integration
08:57:09 <jose_lausuch> I need to leave
08:57:09 <ollivier> sure. Third party should offer "real" entry points
08:57:10 <morgan_orange> in the first release we did not refuse any code...but we could be more strict on the interface now as we reached a critical mass
08:57:27 <morgan_orange> the idea is to ensure some sustainability and maintanbility over time..
08:57:45 <morgan_orange> ok let's continue off line
08:57:57 <morgan_orange> but framework + functest introduction/vision look fine
08:58:07 <morgan_orange> if you have any other ides => mail (deadline 27th...)
08:58:27 <morgan_orange> #info discussion on CFP Beijing
08:58:49 <morgan_orange> #info presentation on Functest (introduction/vision) + framework evolution and interface with third party to be initiated
08:59:09 <morgan_orange> #action morgan_orange jose_lausuch prepare short summary and share with Functest contributors before the 27th
08:59:19 <morgan_orange> #topic Auphrates: what's next
08:59:31 <morgan_orange> #info connected to possible topics, we initiated a wiki page to discuss of Euphrates
08:59:48 <morgan_orange> #link https://wiki.opnfv.org/display/functest/Functest+Euphrates+page
09:00:19 <morgan_orange> it is a draft page, but feel free to put any point you would like to discuss, we will start a new cycle, any idea, even the craziest one, is welcome
09:00:24 <morgan_orange> #topic AoB
09:00:34 <morgan_orange> it is time any other info you would like to share today?
09:00:37 <MatthewLi> have one suggestion.....
09:00:50 <morgan_orange> go ahead MatthewLi
09:01:18 <MatthewLi> usually for dovetail, there's a need to run offline, so functest had better to support offline...
09:02:15 <tallgren> We have a discussion today about Danube release in the TSC meeting. It would be good to have the Functest PoV in the meeting, like what is still needed to release Danube on March 27th
09:02:31 <MatthewLi> and isolated with CI as much as possible...can run just like an independent 3 rd party tool
09:02:44 <morgan_orange> tallgren: I will be there
09:02:49 <tallgren> +1
09:03:20 <morgan_orange> tallgren: I told david that an extra week will not be luxory...as we reached the MS3 intiailly planned mid january only last week end... :)
09:03:56 <tallgren> Yes, and we lost 2 weeks beginning of this year
09:04:00 <morgan_orange> MatthewLi: Ok got your point, you can add it in the wiki, I think we already had also this discussion, some cases can be managed offline
09:04:10 <morgan_orange> not all
09:04:12 <MatthewLi> morgan_orange: thx
09:04:28 <morgan_orange> but severla ones, regarding what is run by dovetail today (vping + refstack) I thin k it is OK
09:04:38 <tallgren> MatthewLi: sorry to speak at the same time, time was running out...
09:04:49 <MatthewLi> tallgren: that's Ok :)
09:04:52 <morgan_orange> for the VNF it is more complex due to lots of dependencies => the docker will be much too fat
09:05:28 <morgan_orange> ok any other question for my last interim PTL meeting?
09:05:41 <ollivier> no. Great job!
09:05:46 <boucherv> no
09:05:55 <hideyasu> no thanks
09:05:55 <HelenYao> morgan_orange: great job
09:06:21 <morgan_orange> but jose_lausuch is now fully wise ....
09:06:28 <morgan_orange> as he is married
09:06:33 <LindaWang> morgan_orange: great job. :)
09:07:01 <MatthewLi> jose_lausuch: congrats
09:07:02 <morgan_orange> with no CI during 8 days it is not difficult :)
09:07:47 <morgan_orange> ok see you next week and enjoy Danube https://www.youtube.com/watch?v=_CTYymbbEL4
09:07:51 <morgan_orange> #endmeeting