08:00:24 <morgan_orange> #startmeeting Functest weekly meeting November 24th
08:00:24 <collabot> Meeting started Tue Nov 24 08:00:24 2015 UTC.  The chair is morgan_orange. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:00:24 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:00:24 <collabot> The meeting name has been set to 'functest_weekly_meeting_november_24th'
08:00:33 <morgan_orange> #info Morgan Richomme
08:00:35 <viktor_nokia> #info Viktor Tikkanen
08:00:39 <juhak> #info Juha Kosonen
08:01:06 <Qinglong> #info Qinglong Lan
08:01:09 <morgan_orange> #info agenda https://wiki.opnfv.org/functest_meeting
08:01:16 <morgan_orange> let's start
08:01:26 <morgan_orange> #topic Action point follow-up
08:01:35 <morgan_orange> #link http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2015/opnfv-testperf.2015-11-17-10.02.html
08:02:08 <morgan_orange> #info action 1 => not possible to grant access to LF POD2, LF POD 2 is considered as a production CI lab, the access is restricted
08:02:35 <morgan_orange> #info access to be found on dev labs, Orange shall be able to grant one access to  one of our lab
08:03:02 <morgan_orange> #info action2) => wiki updated https://wiki.opnfv.org/functextnexttaks
08:03:27 <morgan_orange> #info I added a table showing the dependencies towards Doctor, Copper, ONOSFW, OVNO, PolicyTest and Promise
08:03:31 <xiaoguang> #info XiaoguangLi
08:04:21 <morgan_orange> #info seen from the Summit some projects are already mature => question of integration, some projects not so clear and the availability (or not) of test suites to be integrated in functest for B release
08:04:55 <morgan_orange> #info action 3) work on Orange POD planned tomorrow (jumphost connected, env ready..)
08:05:13 <morgan_orange> #info action 4) JIRA updated and Sprint 3 started
08:05:43 <morgan_orange> #info action 5) viktor_nokia completed Tempest backlog we will have a dedicated topis later
08:06:11 <morgan_orange> #info action 6) discussion on the syntezis of functest for the dashboard...let's discuss it also in a separate topic
08:06:28 <morgan_orange> #topic B release Sprint 3
08:06:43 <morgan_orange> #info Jira Spint 3 started: https://jira.opnfv.org/secure/RapidBoard.jspa?rapidView=59
08:07:15 <morgan_orange> #info transistion Sprint, still some code to be produced (especially ODL test suite reformating) but first tasks on intgeration
08:07:58 <morgan_orange> #info LF POD 2 (Fuel) ready, jose should work on the integration with Apex, Mei Mei with Huawei/Compass and myself should discuss with joid team
08:08:16 <morgan_orange> #info reminder for B release Functest success criteria => running all our suites on the 4 env
08:08:38 <morgan_orange> welcome jose_lausuch :)
08:09:17 <morgan_orange> #topic status on Onos
08:09:23 <morgan_orange> Qinglong: go ahead
08:09:27 <Qinglong> #info Huawei-POD is ready now, Need some time to debug with it
08:09:43 <jose_lausuch> hi sorry for the late :)
08:09:43 <Qinglong> #info ML3 is under working ,we are writing Testcases, about 3 weeks to finish scripts
08:09:45 <morgan_orange> is it Huawei US or Huawei china lab?
08:09:50 <jose_lausuch> #info Jose Lausuch
08:09:52 <Qinglong> china first
08:09:54 <morgan_orange> ok
08:10:02 <Qinglong> then use USA
08:10:07 <morgan_orange> you know who is the lab contact?
08:10:16 <morgan_orange> I assigned Mei mei to run the complete functest
08:10:18 <Qinglong> Mei mei...
08:10:24 <morgan_orange> is she the right contactN
08:10:27 <morgan_orange> ?
08:10:32 <Qinglong> yes
08:10:39 <Qinglong> this morning we had a call
08:10:39 <morgan_orange> ok
08:10:57 <Qinglong> ML3 is under working
08:11:14 <morgan_orange> #info Mei Mei assigned to run functest suite on Huwaei target lab (in china First then US)
08:11:15 <Qinglong> Need some time to write cases & scripts
08:11:24 <xiaoguang> Yes ,I'm now preparing the testing cases and scripts of L3
08:11:30 <morgan_orange> cool
08:12:06 <morgan_orange> i initiated the sprint yesterday for 3 weeks so it is may be a bit short for comletion but shall not be far
08:12:13 <Qinglong> that's all. And we may ask some help we prepare CI
08:12:36 <morgan_orange> of course do not hesitate to ask for help regarding CI integration
08:12:58 <morgan_orange> the goal is to include ONOS as ODL today and to be able to run on any of the installer providing an ONOS configuration
08:13:13 <morgan_orange> CI will be tricky and fun...
08:13:36 <morgan_orange> as mentioned CI work initiated in Sprint 3 but will be more important in Sprint 4 with docs...
08:13:54 <morgan_orange> BTW, I can also help you to declare Onos testcase in the DB
08:13:56 <jose_lausuch> sprint 4 shouldnt have that much of coding
08:14:11 <morgan_orange> if you have already the title of the testcases
08:14:19 <xiaoguang> good
08:14:20 <morgan_orange> i think we could declare only Onos as 1 suite
08:14:40 <morgan_orange> #info possibility to add ONOS as Functest testcase in DB
08:14:42 <morgan_orange> #link https://wiki.opnfv.org/brahmaputra_testing_page
08:14:59 <morgan_orange> #info B testing updated
08:15:00 <jose_lausuch> that would be great
08:15:24 <morgan_orange> #info table to be updated still (with results and dashboard ready status)
08:15:39 <morgan_orange> #info we planned to discuss dashboard and DB during next thursday meeting
08:16:01 <morgan_orange> #action morgan_orange xiaoguang declare ONOS suite in the DB and update wiki accordingly
08:16:15 <morgan_orange> any question on ONOS?
08:16:58 <morgan_orange> #topic tempest
08:17:15 <morgan_orange> #info Viktor updated backlogs https://jira.opnfv.org/browse/FUNCTEST-70
08:18:00 <morgan_orange> #info last 9 errors analyzed by viktor_nokia
08:18:25 <viktor_nokia> I will probably try to run those failed cases one by one in our setup
08:18:27 <morgan_orange> #info most of the errors seem linked to ODL
08:18:46 <morgan_orange> #info interesting to see the difference with ONOS...
08:19:04 <morgan_orange> we do not have also a reference without SDN controller
08:19:14 <morgan_orange> as far as I know it is not considered as a target configuration
08:19:25 <morgan_orange> could be interested to compare...
08:19:46 <morgan_orange> I am a bit surprised by the error related to quota
08:19:49 <jose_lausuch> that would require changing neutron.conf
08:19:54 <jose_lausuch> maybe not so complex
08:20:07 <morgan_orange> no but an additional config to manage
08:20:34 <morgan_orange> at least on dev labs we shall be able to run the suites with SDN controllers just to compare
08:20:40 <jose_lausuch> yes
08:20:43 <jose_lausuch> regarding quota
08:20:52 <jose_lausuch> that happens sometimes if we have run tests before
08:20:58 <jose_lausuch> and havent cleaned them
08:21:10 <morgan_orange> hmm
08:21:25 <morgan_orange> here as far as I remember it was the suite run automatically after a fresh install
08:21:25 <jose_lausuch> but it shouldnt happen when running after a fresh installation
08:21:36 <jose_lausuch> ok
08:21:39 <morgan_orange> so it should not be the case...
08:21:44 <jose_lausuch> then, yes
08:21:55 <viktor_nokia> quota errors could be also caused by resources which were not freed by previously failed test cases (in the same suite)
08:22:29 <morgan_orange> #info discussion on quota errors: strange as they occur after a fresh install and quota had been already extended for Arno
08:22:40 <morgan_orange> #info viktor_nokia "quota errors could be also caused by resources which were not freed by previously failed test cases (in the same suite)"
08:23:19 <morgan_orange> #action viktor_nokia (when a ev lab will be available) test the Tempest suite manually, check quota before the test
08:24:12 <morgan_orange> regarding "subnet net04_ext__subnet of external network net04_ext is visible for the user (could it be so that the user has admin rights for some reason?)"
08:24:47 <morgan_orange> could be linked to the way this default subnet  is created
08:25:01 <morgan_orange> jose_lausuch: it is created by the Fuel installer, not Functest, right?
08:25:07 <jose_lausuch> yes
08:25:09 <jose_lausuch> only in fuel
08:25:14 <jose_lausuch> foreman was different
08:25:17 <viktor_nokia> I checked generated tempest.conf
08:25:19 <jose_lausuch> and apex/compass I guess so
08:25:20 <viktor_nokia> [identity] username = admin password = octopus tenant_name = admin alt_username = admin
08:25:35 <jose_lausuch> tempest uses admin, right?
08:25:49 <morgan_orange> yes
08:25:54 <viktor_nokia> alt_tenant_name = admin as well
08:25:57 <jose_lausuch> and tempest creates its own nets
08:27:04 <morgan_orange> I do not know if it is easy to configure the test to create a different tenant (not admin)
08:27:14 <morgan_orange> create a user + a tenant ?
08:27:18 <jose_lausuch> I guess tempest.conf?
08:27:24 <morgan_orange> for vIMS Valentin created a dedicated tenant
08:27:57 <morgan_orange> OK let's see also if we can progress on that when dev labs will be available
08:28:06 <jose_lausuch> well, you need first a tenant (project) and then as many users as you want (normally 1) for that tenant/project
08:28:21 <morgan_orange> #info error seems linked to configuration , subnet created by fuel is visible, it should not be the case
08:28:42 <morgan_orange> #info discussion on the opportunity to change config in order to avoir running everything as admin, in admin tenant
08:29:02 <morgan_orange> #info subnet mentioned only created in Fuel, not the case with other installers a priori
08:29:11 <morgan_orange> #info so wait and see next run also on other labs to compare
08:29:21 <morgan_orange> #info and see if it is not a Fuel specific error
08:29:46 <morgan_orange> it seems there is also an error linked to the fact that we are on Juno
08:30:03 <morgan_orange> subnetpool-create not supported
08:30:11 <jose_lausuch> mmm that was maybe run on Juno
08:30:23 <jose_lausuch> but the target is Kilo or even Liberty
08:30:29 <morgan_orange> yes
08:30:35 <morgan_orange> so wait and see as well
08:30:52 <morgan_orange> also an issue linked to floatting IP
08:31:35 <viktor_nokia> it caused probably by following row from generated tempest.conf: default_network = 172.30.10.0/24
08:31:54 <viktor_nokia> which is the same as external network
08:32:02 <jose_lausuch> #info tempets.conf to be improved
08:32:34 <jose_lausuch> I thought rally genconf was in charge of that
08:33:34 <morgan_orange> it is the case, but there are probably someway by configuration to modify the target tempest.conf (as we used to faced for the keystone v3 tests) maybe some work to be done upstream in rally
08:34:29 <jose_lausuch> ok
08:34:31 <morgan_orange> #action viktor_nokia test (when dev lab ready) the influence of tempest.conf change into Tempest, and see if upstream patch may be created on rally to fix the issues that would be solved by the config change
08:34:53 <morgan_orange> anyway thanks viktor_nokiafor the backlogs, it is better to see them here rather than in a wiki :)
08:35:22 <morgan_orange> 9 errors assuming taht 2 are clearly identified as ODL, 2 related to quotas and the other maybe due to ODL, misconfiguration
08:35:33 <morgan_orange> I think it is not bad (it was worst in arno.. :))
08:35:42 <jose_lausuch> yes
08:35:45 <morgan_orange> any question regarding Tempest?
08:35:55 <jose_lausuch> nop
08:36:11 <viktor_nokia> what is the target test suite for tempest?
08:36:18 <viktor_nokia> smoke? full?
08:36:25 <morgan_orange> smoke shall be the minimu
08:36:26 <viktor_nokia> or customized?
08:36:34 <morgan_orange> customized will make sense
08:36:51 <jose_lausuch> we can also run full
08:36:58 <jose_lausuch> but that would give a lot of errors maybe
08:37:03 <morgan_orange> yes we can run
08:37:06 <jose_lausuch> as soon as we have smoke 100% working
08:37:13 <jose_lausuch> we can try the full, but...
08:37:18 <jose_lausuch> not sure if its worth it
08:37:28 <viktor_nokia> customized would be a file with individual test case list
08:37:33 <jose_lausuch> yes
08:37:38 <morgan_orange> we imagined that for Arno
08:38:00 <jose_lausuch> I did a list
08:38:06 <jose_lausuch> for smoke
08:38:16 <jose_lausuch> but it was slightly differnt for each installer
08:38:23 <morgan_orange> for me it will be a good option to extend smoke without going to full which will triggers lots of errors
08:38:32 <jose_lausuch> +1
08:38:52 <morgan_orange> that is another point, we must remain in our test as installer agnostic as possible
08:39:06 <morgan_orange> we must not create a customized tempest suite for apex/fuel/compass/joid
08:39:20 <morgan_orange> we must garantee the common suite for all the installers
08:39:27 <viktor_nokia> fully agree
08:39:34 <jose_lausuch> yes, thats right
08:39:47 <morgan_orange> viktor_nokia: can I action you the study of the creation of a customized OPNFV tempest suite?
08:40:01 <viktor_nokia> OK
08:40:19 <morgan_orange> #action viktor_nokia the opportunity to cretae a customized , installer & controller agnostic, OPNFV Tempest suite
08:40:30 <morgan_orange> it will be the same for Rally
08:40:52 <morgan_orange> for the moment we just merged the different scenarios of the different modules, it will probably make sense to create one OPNFV scenario
08:40:58 <morgan_orange> juhak: what do you think?
08:41:21 <juhak> I agree
08:42:01 <morgan_orange> Ok I can actio nyou too then :)
08:42:24 <juhak> ok
08:42:27 <morgan_orange> #action juhak study the creation of an OPNFV rally scenario
08:42:44 <morgan_orange> I think both action are covered by the current JIRA, no need to additional admin stuff
08:43:03 <morgan_orange> I suggets we spend some time next week to focus on Rally
08:43:05 <jose_lausuch> great
08:43:19 <morgan_orange> #topic Discussion on the summary of Functest for dashboard
08:43:29 <morgan_orange> it was a action from last week
08:43:39 <morgan_orange> in the dashboard we shall provide an overall status
08:43:58 <morgan_orange> what will be our criterai for B-Release seen from the dashboard
08:44:34 <jose_lausuch> I would say
08:44:43 <jose_lausuch> everythink working ! :)
08:44:47 <viktor_nokia> total/passed numbers for each project at least?
08:44:58 <morgan_orange> yes
08:45:36 <morgan_orange> I would say vPING 100% OK,  Tempest & Rally (errors shall be documented) but we expect at least 90% for Tempest smoke suite
08:46:03 <jose_lausuch> if we get behriluim we might have 100% smoke
08:46:06 <morgan_orange> for SDN controllers suites, as they are defined by us I would say 100% (but for ODL last time we got 3 errors that were documented bue to ODL)
08:46:38 <morgan_orange> and for me vIMS the criteria will be the ability to deploy orchestrator, VNF and run tests
08:46:47 <jose_lausuch> as arno, we might get diffrent results for diffrent installers
08:46:51 <morgan_orange> but we do not really care of the results of the tests as they have not be tuned
08:48:19 <morgan_orange> basically what shall we put in the landing page https://www.opnfv.org/opnfvtestgraphs/summary for functest
08:48:29 <morgan_orange> to provide automatic green light from CI
08:48:53 <jose_lausuch> ok
08:48:58 <morgan_orange> are you Ok with
08:49:05 <morgan_orange> - number of test suites
08:49:21 <morgan_orange> and independantly from the installers
08:49:26 <morgan_orange> - vPing 100% OK
08:49:36 <viktor_nokia> so this page is the dashboard we are talking about?
08:49:51 <morgan_orange> yes
08:49:56 <morgan_orange> this page is the prototype
08:50:05 <morgan_orange> from jenkins, results are pushed into DB
08:50:20 <morgan_orange> then LF portal (this page) get info from the DB to build the dashboard
08:50:40 <morgan_orange> I created a "virtual" use case called status that could be used to give the overview of the project
08:50:49 <morgan_orange> instead of Test cases (3): vPing (functest), Tempest (functest), Ping (yardstick)
08:50:51 <morgan_orange> in the landing page
08:51:27 <morgan_orange> Marion from LF shall also add filters on the existing page to be able to compare the results per POD if we consider the project view
08:51:31 <morgan_orange> https://www.opnfv.org/opnfvtestgraphs/per-test-projects
08:52:00 <viktor_nokia> BTW, foreman tempest number of failures (~40) corresponds with failure numbers in our foreman setup
08:52:18 <morgan_orange> we used to have more errors on foreman as well
08:52:34 <morgan_orange> not so many differences
08:52:39 <morgan_orange> as far as i remember
08:52:47 <morgan_orange> but now tim migrated to RDO
08:52:52 <morgan_orange> foreman will no more supported
08:53:00 <viktor_nokia> ???
08:53:02 <morgan_orange> so no need to spend time on troubleshouting
08:53:13 <viktor_nokia> How about Apex?
08:53:24 <morgan_orange> Apex is based on RDO, which is based on OOO
08:53:37 <morgan_orange> that is a different approach (no more foreman)
08:53:48 <viktor_nokia> so Foreman is completely abandoned?
08:53:57 <morgan_orange> yes
08:54:12 <morgan_orange> that is my understanding
08:54:22 <jose_lausuch> I think so, yes
08:54:33 <jose_lausuch> they are not longer working on it
08:54:41 <jose_lausuch> all the efforts go to apex
08:54:47 <jose_lausuch> and we will be able to run functest soon
08:54:54 <jose_lausuch> for now, just virtualized
08:54:57 <morgan_orange> regarding the difference in arno between the installer we have this view
08:54:58 <morgan_orange> https://www.opnfv.org/opnfvtestgraphs/per-installers
08:55:50 <morgan_orange> and last run with foreman done on LF POD was not far from 40 errors
08:56:19 <morgan_orange> for 100 tests run (compared to 110 tests for 14 erros on Fuel)
08:56:33 <morgan_orange> so back to my summary
08:57:12 <morgan_orange> https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api/dashboard/functest2Dashboard.py
08:57:22 <morgan_orange> for the status case
08:57:29 <morgan_orange> I planned (but not implemented yet)
08:57:35 <morgan_orange> nb of suites run
08:58:07 <morgan_orange> average tempest failure rate (shall be less than 10%)
08:58:37 <morgan_orange> vPing (must be fully OK)
08:58:54 <morgan_orange> and I suggest that all the controllers suite must be OK (as we manage them - not upstream suite)
08:59:08 <morgan_orange> vIMS, just consider the run in jenkins not the restuls
08:59:09 <jose_lausuch> ok
08:59:31 <jose_lausuch> and rally?
08:59:34 <morgan_orange> #info discussion on the criteria to be automatically displayed in dashboard page for B-release acceptance
08:59:40 <morgan_orange> rally like Tempest
08:59:43 <jose_lausuch> ok
08:59:49 <morgan_orange> may be create only VIM category
09:00:10 <morgan_orange> #link https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api/dashboard/functest2Dashboard.py
09:00:21 <morgan_orange> so we will have very basic lights
09:00:32 <morgan_orange> grenn lights for the VIM => tempest + rally > 90 %
09:00:41 <morgan_orange> green light for the controllers => 100%
09:00:49 <jose_lausuch> ok, lets keep it simple
09:00:51 <jose_lausuch> I agree
09:00:51 <morgan_orange> green light basic => vPing 100%
09:01:04 <morgan_orange> green light VNF deployment => deployment vIMS
09:01:09 <morgan_orange> 4 criteria
09:01:09 <jose_lausuch> let me info that
09:01:18 <morgan_orange> coreesponding to the 6 or 7 suites
09:01:20 <jose_lausuch> #info grenn lights for the VIM => tempest + rally > 90 %
09:01:27 <jose_lausuch> #info green light for the controllers => 100%
09:01:30 <morgan_orange> 2 for VNFs + 2 for VIMs + N for Controllers
09:01:35 <jose_lausuch> #info green light basic => vPing 100%
09:01:41 <jose_lausuch> #info green light VNF deployment => deployment vIMS
09:01:47 <morgan_orange> ok
09:01:59 <morgan_orange> #action morgan_orange implement status for dashboard
09:02:04 <morgan_orange> #topic AoB
09:02:07 <morgan_orange> we are already late
09:02:09 <jose_lausuch> I have something here :)
09:02:10 <jose_lausuch> yes
09:02:11 <morgan_orange> sut some information
09:02:12 <jose_lausuch> let me info quick
09:02:23 <morgan_orange> #info change in Functest team
09:02:45 <morgan_orange> #info -3 (HP, metaswitch) and +2 (Cisco, HP)
09:02:54 <jose_lausuch> #info Fuel already integrated OpenContrail, I have a contact person from Juniper to see what test to run. Work to be started this week.
09:02:56 <morgan_orange> #info will send a mail for formal nomination
09:03:01 <jose_lausuch> oops, I should wait
09:03:36 <morgan_orange> jose_lausuch: cool so maybe update the dependeincis towards OVNO in https://wiki.opnfv.org/functextnexttaks
09:04:21 <morgan_orange> #info ODL extension, Peter and amaged__ should work on this, refactoring to be planned. For B-Release cherry pick of ODL tests (GBP + integration tests) for C release study to automate the integration of upostream tests to be planned
09:04:24 <jose_lausuch> ok
09:04:31 <morgan_orange> any othe rinfo you would like to share?
09:04:34 <jose_lausuch> yes
09:05:14 <morgan_orange> ok have a good week and enjoy Sprint #3
09:05:18 <morgan_orange> #endmeeting