08:00:27 <morgan_orange1> #startmeeting Functest weekly meeting March 7th
08:00:27 <collabot`> Meeting started Tue Mar  7 08:00:27 2017 UTC.  The chair is morgan_orange1. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:00:27 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:00:27 <collabot`> The meeting name has been set to 'functest_weekly_meeting_march_7th'
08:00:33 <morgan_orange1> #topic call role
08:00:38 <morgan_orange1> #info morgan Richomme
08:00:55 <juhak> #info Juha Kosonen
08:01:11 <hideyasu> #info Hideyasu Hayashi
08:01:21 <LindaWang> #info Linda Wang
08:01:24 <morgan_orange1> #info agenda for today https://wiki.opnfv.org/display/functest/Functest+Meeting
08:01:41 <CristinaPauna> #info Cristina Pauna (Enea)
08:01:53 <morgan_orange1> any topic you want to add to the agenda?
08:02:20 <morgan_orange1> #topic action points follow up
08:02:33 <morgan_orange1> 8 action points from last week
08:02:40 <morgan_orange1> #info AP1: ollivier add a wiki page about review best practices
08:02:46 <morgan_orange1> ollivier: you confirm, it is not started?
08:03:00 <morgan_orange1> #info not started (but not the priority regarding the release)
08:03:10 <morgan_orange1> #action ollivier add a wiki page about review best practices
08:03:19 <morgan_orange1> #info AP2: jose_lausuch update documentation for: removed healthcheck, added tempest tests
08:03:26 <morgan_orange1> #info done, patch submitted and structure agreed with other test projects
08:03:35 <morgan_orange1> #info AP3: contact Refstack PTL to see why only 104 / 244 tests are run
08:03:41 <morgan_orange1> #info done: seems negative tests are not executed
08:03:43 <shuya_ool> #info shuya(ool)
08:03:47 <morgan_orange1> #info work on using refstack client rather than starting tempest suite from Tempest/rally to be done
08:03:54 <morgan_orange1> #info AP4: LindaWang troubleshoot securtiy_scan issue
08:04:00 <morgan_orange1> #info done, see next topic discussion
08:04:11 <morgan_orange1> LindaWang: we will discuss in next topic, but I consider it is done :)
08:04:20 <morgan_orange1> #info AP5: morgan_orange create a patch to exclude Doctor temporarily + add Doctor team in review
08:04:26 <morgan_orange1> #info done, patch will be abandonned - first Doctor run OK on fuel
08:04:34 <morgan_orange1> #info AP6: re-enable promise on the dashboard
08:04:42 <morgan_orange1> #info done
08:04:50 <morgan_orange1> #info AP7: morgan_orange sync joid / Steve/ Narinder for SNAPS errors on joid scenarios
08:04:55 <morgan_orange1> #info partially done, os-nosdn-nofeature scenario is now fine, still bug in odl and open questions on lxd scenarios / SNAPS (see next sections)
08:05:02 <morgan_orange1> #info AP8: jose_lausuch ask for a pod for weekly loops
08:05:10 <morgan_orange1> #info done....see next topic
08:05:34 <morgan_orange1> any question on teh action points? some topic will be discussed in the next topics (weekly , security scanning,..)
08:06:06 <ollivier> morgan_orange1: yes. pending....
08:06:15 <morgan_orange1> if you are OK I suggest to address these topics before the status
08:06:23 <morgan_orange1> no objection?
08:06:36 <morgan_orange1> #topic Security scanning: testbase versus feature base for python based feature projects
08:07:04 <morgan_orange1> LindaWang: fixed the path issues regarding security scanning, but in the review we saw that we were about to re do what we tried to eliminate...
08:07:14 <morgan_orange1> it means python => bash => python
08:07:22 <morgan_orange1> as mentione din the comment of the patch
08:07:55 <morgan_orange1> #link https://gerrit.opnfv.org/gerrit/#/c/29703/
08:08:26 <morgan_orange1> this is due to the use of Featurebase and extension of testbase adressing the feature projects that at the beginning were only dealing with bash scripts..
08:08:33 <morgan_orange1> it is not the case for security_scanning
08:09:29 <morgan_orange1> so 2 options: use TestBase and not featureBase (recoding needed) or accept this python/bash/python for security scanning
08:09:55 <morgan_orange1> we are now between MS6 and MS7
08:10:00 <morgan_orange1> so close to the release
08:10:29 <morgan_orange1> what is your view?
08:10:56 <ollivier> why should we call cd? or source?
08:11:25 <morgan_orange1> #info security scanning patch will fix security scan problem on apex results. However it reintroduced a python => bash => python we tried to exclude
08:11:56 <morgan_orange1> we call cd because we call the python files from security scanning repo (not our code, it is an external feature project)
08:13:00 <morgan_orange1> but that is the only external feature project called through python command (all the other feature projects are call through bash or node command)
08:14:02 <ollivier> yes but you could switch to the related python os library
08:14:05 <LindaWang> There are other test cases like  SFC, bgpvpn .
08:14:23 <morgan_orange1> quick way => merge the patch and reconsider the problem in E (way to call external feature project from feature base)
08:15:00 <morgan_orange1> LindaWang: you are right..
08:15:07 <morgan_orange1> https://git.opnfv.org/functest/tree/functest/opnfv_tests/features/sdnvpn.py
08:15:18 <morgan_orange1> https://git.opnfv.org/functest/tree/functest/opnfv_tests/features/odl_sfc.py
08:15:31 <morgan_orange1> Ok so I suggest to merge and to reconsider this for E
08:15:36 <morgan_orange1> any objection?
08:15:41 <LindaWang> agree
08:15:44 <ollivier> morgan_orange1: I disagree.
08:15:48 <SerenaFeng> #agree
08:16:00 <ollivier> morgan_orange1: it's quite simple to write it in a python way
08:16:43 <morgan_orange1> you mean modify featureBase?
08:17:00 <ollivier> https://gerrit.opnfv.org/gerrit/#/c/29703/2/functest/opnfv_tests/features/security_scan.py
08:17:12 <morgan_orange1> if we do it for security_scanning we should do it for sfc and sdnvpn to be consistant
08:18:43 <ollivier> I don't understand the point.
08:19:16 <morgan_orange1> ok we can discuss it offline
08:20:00 <morgan_orange1> #action morgan_orange ollivier clarify python way for feature projects inhering feature based - see impact on sfc, sdnvpn
08:20:19 <morgan_orange1> #topic Weekly jobs (vrouter, orchestra_ims, cloudify_ims)
08:20:34 <morgan_orange1> #info first weekly job created on fuel/os-nosdn-nofeature-ha scenario
08:20:53 <morgan_orange1> #info it is now possible to launch manually a weekly job , mainly to test the VNF
08:21:07 <morgan_orange1> #link https://build.opnfv.org/ci/job/functest-fuel-baremetal-weekly-master/
08:21:28 <morgan_orange1> #info first attempt => problem in cloudify_ims preventing vrouter and orchestra_ims to be run
08:21:51 <morgan_orange1> #aciton valentin troubleshoot the issue, at least it shall not prevent the other tests to be run
08:22:09 <morgan_orange1> hideyasu: in theory vrouter should be run from CI after cloudify_ims
08:22:14 <ollivier> morgan_orange1: you could easily override run() for security scan
08:22:32 <hideyasu> ok thank you.
08:22:36 <morgan_orange1> I will see with valentin, if needed we can disable the case to be sure yours is run
08:22:45 <morgan_orange1> same for orchestra
08:23:10 <ollivier> morgan_orange1: I think run should be unimplemented by default as TestCaseBase.
08:23:38 <morgan_orange1> #info orchestra torubleshooting done on virtual server, but it seems that issues may be due to resources shortage on virtual env, tests need to be done on baremetal to confirm
08:24:21 <SerenaFeng> ollivier run() support to be overrided in feature_base
08:24:49 <morgan_orange1> weekly run can be triggered though jenkins interface, when launched, it waits for available resources
08:25:33 <morgan_orange1> #info weekly runs can be triggered manually though jenkins interface (today one installer/1 scenario)
08:25:48 <SerenaFeng> #link https://gerrit.opnfv.org/gerrit/#/c/28609/, jose do it in this patch
08:26:15 <hideyasu> morgan: I found one issue for vrouter. so I will push patch today.
08:26:23 <hideyasu> https://jira.opnfv.org/browse/FUNCTEST-750
08:26:38 <morgan_orange1> #info issue found on vrouter, patch will be submitted today
08:26:45 <morgan_orange1> #link https://jira.opnfv.org/browse/FUNCTEST-750
08:27:18 <morgan_orange1> #topic Troubleshooting status
08:27:29 <morgan_orange1> #info mail sent yesterday
08:27:38 <morgan_orange1> #action morgan_orange => share the mail with scenario owners (any alias?)
08:27:45 <morgan_orange1> #info bug fix provided for Doctor, successfully tested on fuel
08:27:49 <morgan_orange1> #info status apex
08:27:54 <morgan_orange1> #link http://testresults.opnfv.org/reporting/functest/release/master/index-status-apex.html
08:28:00 <morgan_orange1> #info new run this morning (48h without CI run
08:28:26 <MatthewLi> #info Jun Li
08:28:41 <morgan_orange1> #info security scanning: we know how to solve but discussion to solve it properly ...(see previous topic that needs follwo up by mail)
08:28:48 <morgan_orange1> #info considering Doctor/Security scanning OK => 1 scenario OK for Danube nosdn-nofeature-ha
08:28:56 <morgan_orange1> #info main issues: Tempest_smoke (odl_l3, bgpvpn), SNAPS (gluon), vping_ssh (bgpvpn)
08:29:17 <morgan_orange1> #info bgpvpn vping_ssh pb could be linked to last ODL version? (see Manuel's mail)
08:29:24 <morgan_orange1> #info fds scenario in CI but failing SNAPS so not see in the reporting page (SNAPS patch provided to supported huge page)
08:29:43 <morgan_orange1> #action juhak : give feedback on tempest_smoke error for apex scenario
08:29:53 <morgan_orange1> #info status compass
08:29:59 <morgan_orange1> #link http://testresults.opnfv.org/reporting/functest/release/master/index-status-compass.html
08:30:05 <morgan_orange1> #info 2 scenarios OK for release: nosdn-nofeature-ha and odl_l2-nofeature-ha
08:30:12 <morgan_orange1> #info main issues: tempest_smoke & rally_sanity (odl_l3), ODL (odl_l3), vping_ssh (ocl)
08:30:28 <morgan_orange1> #action juhak have a look at tempest&rally issue on odl_l3 scenario (compass)
08:30:36 <morgan_orange1> #action ollivier see what is wrong on odl_l3 scenario
08:30:44 <morgan_orange1> #info status fuel
08:30:52 <morgan_orange1> #link http://testresults.opnfv.org/reporting/functest/release/master/index-status-fuel.html
08:31:07 <morgan_orange1> #info 6 scenarios ~ OK for Release (Doctor exluded until we got the number of iterations): nosdn-ov__kvm_dpdk_noha, nosdn-ovs (ha and noha), nosdn_nofeature (ha and noha)
08:31:13 <morgan_orange1> #info CI crash when launching barometer suite
08:31:19 <morgan_orange1> #info issues vping_ssh (SFC), Tempest (odl_l3, odl_l2n bgpvpn), rally_sanity (kvm), SNAPS (kvm), bgpvpn (bgpvpn)
08:31:58 <morgan_orange1> #action juhak have a look at tempest&rally issues on odl & kvm scenarios (fuel)
08:32:04 <morgan_orange1> #info status joid
08:32:10 <morgan_orange1> #link http://testresults.opnfv.org/reporting/functest/release/master/index-status-joid.html
08:32:16 <morgan_orange1> #info 2 scenarios OK (without considering Doctor now) for Danube nosdn-nofeature (ha and noha)
08:32:22 <morgan_orange1> #info main issues: vping_ssh (odl), rally_sanity (lxd), SNAPS (lxd)
08:32:28 <morgan_orange1> #info k8 scenarios no tested => rto be done in E
08:32:43 <morgan_orange1> #action juhak see rally_sanity issue on lxd scenario (joid)
08:33:29 <morgan_orange1> regarding their issue on odl scenario, it could also be linked ot problme reported in Boron SR2? => vping_ssh is not working
08:34:03 <morgan_orange1> narinder asks for support but I think it is a work installer / upstream(ODL)
08:34:21 <morgan_orange1> #action steve and morgan_orange, see if SNAPS smoke is compatible with lxd (probably not...) if not exclude it
08:34:27 <morgan_orange1> any question on the status?
08:35:08 <morgan_orange1> sorry juhak to action you so much..but there are not so many scenarios... :)
08:35:31 <juhak> morgan_orange1: no problem
08:35:48 <morgan_orange1> #info please note that a new testcase has been introduced on os-nosdn-nofeature-ha scenario called tempest devcore, it corresponds to the work mentioned by Jose last year (working with Openstack on ref VNF suite)
08:35:53 <morgan_orange1> #undo
08:35:53 <collabot`> Removing item from minutes: <MeetBot.ircmeeting.items.Info object at 0x24629d0>
08:36:02 <morgan_orange1> #info please note that a new testcase has been introduced on os-nosdn-nofeature-ha scenario called tempest devcore, it corresponds to the work mentioned by Jose last year (working with Openstack on ref NFV suite)
08:36:39 <morgan_orange1> if no question I move to next topic...
08:36:52 <morgan_orange1> #topic Danube branching
08:37:11 <morgan_orange1> the branching windows ends on the 10th of March
08:37:21 <morgan_orange1> if we do not do it, it will be done by aric
08:37:51 <morgan_orange1> I suggest to do it, it is OK to do it on the 10th at 2PM CET?
08:38:00 <morgan_orange1> is there any objection?
08:38:09 <SerenaFeng> nope
08:38:16 <joehuang__> hello, multisite functest doesn't work yet
08:38:32 <joehuang__> test not triggered
08:38:57 <morgan_orange1> ok we finish this topic and we may discuss the issue
08:39:19 <joehuang__> hope this could be addressed before branching :)
08:39:28 <morgan_orange1> #action morgan_orange create Danube branch March 10th 2PM CET
08:39:42 <morgan_orange1> #action morgan_orange adapt reporting to include Danube version
08:40:10 <morgan_orange1> the consequences, once the branch is here => fix on master => then cherry pick to Danube..
08:40:16 <morgan_orange1> Danube shall be stable
08:40:29 <joehuang__> ok
08:40:47 <morgan_orange1> we should also finalize the discussion on security scanning /python way before teh 10th...
08:41:00 <morgan_orange1> #topic Multisite
08:41:26 <morgan_orange1> #info last run 27/2
08:41:28 <morgan_orange1> #link https://build.opnfv.org/ci/view/functest/job/functest-compass-virtual-daily-master/1219/console
08:41:43 <joehuang__> hello, Juha said it may caused by
08:41:44 <morgan_orange1> #info multisite case was failing
08:41:46 <joehuang__> The problem seems to relate to the fact there's no UUID defined in kingbird tests and at least according to tempest documentation every test should have: https://docs.openstack.org/developer/tempest/HACKING.html#test-identification-with-idempotent-id .
08:42:13 <juhak> I'm currently checking this
08:42:42 <juhak> got some logs from Goutham too
08:42:53 <joehuang__> multisite tempest cases could be executed successfully manually
08:42:59 <joehuang__> thanks a lot, juhak
08:43:08 <morgan_orange1> #info  multisite tempest cases could be executed successfully manually
08:43:26 <morgan_orange1> #action juhak troubleshooting multisite
08:43:34 <joehuang__> and test cases could be discovered by tempest too
08:43:40 <morgan_orange1> joehuang why is it not luanched anymore in CI?
08:44:07 <morgan_orange1> oops it is
08:44:09 <morgan_orange1> #link https://build.opnfv.org/ci/job/multisite-kingbird-virtual-daily-master/
08:45:01 <morgan_orange1> #info last attempt
08:45:02 <morgan_orange1> #link https://build.opnfv.org/ci/job/functest-fuel-virtual-suite-master/271/console
08:45:02 <joehuang__> it worked previously
08:46:30 <joehuang__> we just restored the kingbird test after this patch merged: https://gerrit.opnfv.org/gerrit/#/c/29003/
08:46:51 <joehuang__> around two weeks ago
08:47:24 <morgan_orange1> I am looking for the trace...I got blue status even if it was failed
08:48:02 <joehuang__> Thanks
08:48:08 <juhak> kingbird cases are on tempest test case list but for some reason not executed
08:48:24 <morgan_orange1> ok let's see what you will find
08:48:34 <morgan_orange1> #topic JIRA
08:48:52 <morgan_orange1> #info I started cleaning the JIRAs
08:48:53 <joehuang__> thank you all.
08:49:14 <morgan_orange1> #action all close the JIRA that are fixed to clarify
08:49:30 <morgan_orange1> #info if task cannot be done for Danube, move them to E
08:49:52 <morgan_orange1> #info JIRA shall reflect some reality...
08:50:20 <morgan_orange1> #itincludes the JIRAs we created for the internships
08:50:48 <morgan_orange1> #info it includes the JIRAs we created for the internships
08:51:57 <morgan_orange1> #info current status: 29 open issues + 62 in the backlogs that are still tagged Danube
08:52:19 <morgan_orange1> it doe snot seem very reasonable 3 weeks before the release :)
08:52:47 <morgan_orange1> #action morgan_orange clean JIRA for Danube
08:52:59 <morgan_orange1> #topic AoB
08:53:05 <morgan_orange1> any topic you want to add
08:53:16 <LindaWang> morgan_orange1 there are some issues about tempest_defcore.
08:53:32 <LindaWang> Jose has checked with PTL of refstack, but some problems are under discussion and trial.
08:53:46 <LindaWang> it is just that negative tests are not run.  The results that defcore run by functest and refstack client is different.
08:53:57 <LindaWang> Matthew Li  from Dovetail and I are trying refstack client now.
08:54:28 <morgan_orange1> LindaWang: this testcase is not really considered for Danube
08:54:45 <morgan_orange1> it is more for E where we will try to have a collaborative approach with OpenStack on this topic
08:54:58 <morgan_orange1> the idea will be to introduce a refstack client
08:55:14 <morgan_orange1> it is currently run only on os-nofeature-nosdn-ha
08:55:20 <MatthewLi> morgan_orange1: on Danube if not wrong, few weeks ago meeting ondovetail...
08:55:23 <morgan_orange1> through Rally/Tempest
08:55:36 <MatthewLi> morgan_orange1: yep run on os-nofeature-nosdn-ha
08:56:08 <morgan_orange1> I did not discuss it fully with Jose, but I think it will be too short to include a new refstack client for Danube
08:56:22 <morgan_orange1> we will run the suite with our current tooling (Rally/Tempest)
08:56:27 <LindaWang> sure
08:56:35 <MatthewLi> morgan_orange1: this tool is very easy to adapt
08:56:39 <morgan_orange1> we do not have the same results and need to analyze why, then include refstack client,...
08:57:20 <morgan_orange1> it is not a question of easiness.. :) but to try to have something stable for the release...
08:57:31 <MatthewLi> morgan_orange1: yep :)
08:57:49 <MatthewLi> will try best to get it stable
08:57:55 <morgan_orange1> another question I have: Serena are the fixes on the testapi now automatically deployed on testresults?
08:58:11 <morgan_orange1> I did not check for the results of rally_sanity
08:58:21 <morgan_orange1> I need to reenable it on reporting
08:59:07 <morgan_orange1> I do not remember if we enable the automatic update from rohit
08:59:27 <morgan_orange1> if not could you enable it or at least deployed the last fixes?
09:00:01 <morgan_orange1> #action SerenaFeng check testapi fixed are included on testresults.opnfv.org, if not apply them and maybe enable automatic mechanism now..
09:00:36 <morgan_orange1> #info tomorrow Test weekly meeting (APAC slot): with a presentation of Kumar work on VNF catalogue. Feel free to join
09:00:44 <morgan_orange1> any other questions/topics for today?
09:01:30 <morgan_orange1> we did not speak about documentation review...let's do it next week
09:01:36 <morgan_orange1> thanks for joining
09:01:40 <morgan_orange1> have a good week
09:01:47 <morgan_orange1> #endmeeting