08:00:17 #startmeeting Functest weekly meeting January 19th 08:00:17 Meeting started Tue Jan 19 08:00:17 2016 UTC. The chair is morgan_orange. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:00:17 Useful Commands: #action #agreed #help #info #idea #link #topic. 08:00:17 The meeting name has been set to 'functest_weekly_meeting_january_19th' 08:00:25 #info Ahmed Maged 08:00:27 #info Viktor Tikkanen 08:00:28 #info Juha Kosonen 08:00:33 #info Morgan Richomme 08:00:40 #info Jose Lausuch 08:00:48 hello everybody 08:01:14 #info agenda: https://wiki.opnfv.org/functest_meeting 08:01:26 #info mainly focusing on B release at Release -2 weeks 08:01:43 #topic action points follow-up 08:01:55 #link http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2016/opnfv-testperf.2016-01-12-08.00.html 08:02:15 #info AP 1: run test onos 08:02:28 May-meimei: lixiaoguang any update ? 08:02:49 #info AP 2: Jira reorganization for odl => done 08:03:23 #info AP 3 => complete onos section in Functest done => pending (May-meimei lixiaoguang) 08:03:31 morgan_orange: hello 08:03:47 #info AP 4: prepare doctor env => done (and also promise and ovno) 08:04:07 #info AP5 jose_lausuch move rally to 0.2.0 => done 08:04:22 #info AP6 morgan_orange creation of next release page => not done, focus on B release 08:04:30 hello May-meimei 08:04:39 morgan_orange: compass+onos test partly ok 08:04:40 did you succeed in running Onos test suite? 08:04:44 cool 08:04:55 vping failed 08:05:07 I saw this morning that 18/18 passed tests on ODL => compass first installer getting this results 08:05:12 May-meimei: can you do a manual vping test? 08:05:16 because of metedata 08:05:23 create 2 vms, ssh to one and ping the other one? manually? 08:05:30 onos can't support that 08:05:48 jose_lausuch: manual vping is ok 08:05:53 we use "userdata" in vping, that was causing problems some time ago until we fixed it 08:06:02 so, in compass we can't use userdata in nova? 08:06:06 it depends on the config, onos already supported vping 08:06:18 onos ? 08:06:18 #topic B release situation 08:06:38 morgan_orange: jose_lausuch: manual vping on compass+onos is ok 08:06:59 #info manual vping on compass+onos is ok but issue during last CI run 08:07:09 jose_lausuch: morgan_orange: but compass+nosdn and compass+odl will be ok 08:07:44 #info new pages created to capture situation for B Release 08:07:53 compass+odl have come other issues 08:08:06 #link https://wiki.opnfv.org/vping_brahmaputra_page 08:08:13 #link https://wiki.opnfv.org/tempest_brahmaputra_page 08:08:19 #link https://wiki.opnfv.org/vims_brahmaputra_page 08:08:30 may be we shall proceed installer by installer 08:08:35 ok 08:08:43 #action amaged__ create odl page for brahmaputra 08:08:53 BTW, I'm completely lost with available scenarios, do we have a complete list of them somewhere? 08:09:14 viktor_nokia: there 2 lists of scenario 08:09:25 1 list = available scenario = jenkins 08:09:39 https://build.opnfv.org/ci/view/OPNFV%20Platform%20CI%20-%20Alternative%20View/ 08:09:45 1 list ) candidate scenario 08:09:54 https://wiki.opnfv.org/brahmaputra_testing_page 08:10:12 this second list is a bit fuzzy because scenario are sometimes described several times 08:10:22 hi ,morgan there are some testcase onosfw does not supported, could we pick them out when we run CI functest? 08:10:39 Zhanghaoyu: what things? 08:10:45 which testcases? 08:10:50 morgan_orange: OK, thanks! 08:10:51 the wiki was used to identify which scenario were needed, once implemented (i.e. jenkins job) => they are removed from the wiki list 08:11:02 so basically our scenario = jenkins scenario 08:11:07 for B release 08:11:28 the convergence on the name of the scenario has been agreed only 1 week ago... 08:11:35 metadata test case 08:11:42 teh format is os-[controller]-[feature]-[mode] 08:12:05 with controller = nosdn (pure neutron), odl_l2, odl_l3, onos, ocl 08:12:12 mode = ha, non ha 08:12:13 we parse [controller] in functest and select the proper test to run 08:12:16 same for feature 08:12:22 et feature = ovs (dpdk), kvm,... 08:13:07 in CI we can put constraints on any test case. With these constraints we build dynamically the list of tests runnable on this scenario 08:13:09 not "non ha" but "noha" 08:13:46 David_Orange: correct, thanks :) 08:13:51 let(s me info some info 08:14:00 the constraints for each test case is defined in config_functest.yaml 08:14:01 #info question on the scenario 08:14:21 #info available scenario = jenkins 08:14:33 #link https://build.opnfv.org/ci/view/OPNFV%20Platform%20CI%20-%20Alternative%20View/ 08:14:42 #info candidate scenario (not yet in jenkins) = wiki 08:14:48 #link https://wiki.opnfv.org/brahmaputra_testing_page 08:14:57 #info for B release we must support available scenario 08:15:18 #info moving from candidate to available is a question between feature & installer projects 08:15:33 #info question on conditions to run the tests 08:15:47 #info mechanism in place in functest, we can put constraint on each testcase 08:16:10 #info list of runnable testcase defined based on the comparison between CI info (scenario, controller) and contraints defined statically in functest 08:16:36 #link https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml 08:16:45 #info at the end of this file you may see the constriants 08:17:12 but the approach is to associate constraints to a testcase not to exclude 08:17:32 so we may say for instance that vPing must work only in apex, fuel, joid and not compass 08:17:32 jose_lausuch: is there any config file for us to config which testcase can not be runed in CI job? 08:17:59 Zhanghaoyu: you just got an answer from morgan_orange :) 08:18:21 Zhanghaoyu: but in your case you would like to exclude a configuration right? 08:19:10 if we consider vping => no constraint so we systematically will try it towards any fresh installation 08:19:21 the system of constraint is based on regex 08:20:06 e.g. promise must be tested on fuel or joid 08:20:15 not sure we can exclude a config 08:20:17 to be checked 08:20:23 back to installer 08:20:27 status on apex 08:20:30 #info status on apex 08:20:47 #link https://build.opnfv.org/ci/view/functest/job/functest-apex-opnfv-jump-1-daily-master/lastSuccessfulBuild/consoleText 08:21:24 issue with vPing as well 08:21:32 morgan_orange: https://build.opnfv.org/ci/view/functest/job/functest-apex-opnfv-jump-1-daily-master/20/consoleFull 08:21:41 lastSuccessfulBuild might be different every time 08:21:43 take the id 08:21:55 ya, vping failed 08:21:57 morgan_orange: https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml in this file right? actually some testcase we dont support now 08:22:02 and we have another small problem 08:22:04 Zhanghaoyu: yes 08:22:17 juhak: viktor_nokia: something strange happens with tempest results 08:22:31 OK Zhanghaoyu let me indicate that offline 08:23:08 ====== 08:23:08 Totals 08:23:08 ====== 08:23:08 Ran: 97 tests in 160.0000 sec. 08:23:08 - Passed: 89 08:23:08 - Skipped: 0 08:23:09 - Expected Fail: 0 08:23:09 - Unexpected Success: 0 08:23:10 - Failed: 8 08:23:11 Sum of execute time for each test: 314.3107 sec. 08:23:23 but the, if you do a rally verify list 08:23:25 you get 08:23:30 Results: {'timestart': '2016-01-1823:24:28.399601', 'duration': 169, 'tests': 113, 'failures': 24} 08:23:34 different numbers 08:23:53 morgan_orange: im a little affraid about whether these testcase take some negative effect on our ONOSFW release 08:24:45 jose_lausuch: if we pick them out it don't have some problems about B release? 08:24:53 Zhanghaoyu: why? apex has also some trouve and inidcate for instance that vIMS case will not be supported 08:25:43 :) just a little worried about it, thank you for your response~ 08:25:54 BTW on apex/onos Zhanghaoyu May-meimei lixiaoguang => ImportError: No module named pxssh 08:26:04 I think that is the error I saw during the week end 08:26:11 seems a library is missing 08:26:49 jose_lausuch: It looks like "setUpClass ... ... FAILED" cases are counted as 1 test/failure in Totals but actually more than one test case is skipped (=failed) because of setUp failure 08:26:53 requirements file must be updated 08:27:17 pexpect must be added to go further with onos test run from CI 08:27:42 jose_lausuch: this is just a theory, I'm not sure 08:27:52 #action May-meimei add pexpect lib in https://git.opnfv.org/cgit/functest/tree/docker/requirements.pip 08:28:05 viktor_nokia: ok.. its a bit confusing 08:28:38 rally is running (we will make a specific topic for rally after installer review) 08:28:54 viktor_nokia: even the number of tests is different 08:29:01 and the time 08:29:02 Ran: 112 tests in 93.0000 sec. 08:29:09 duration': 105, 'tests': 120 08:29:38 regarding Tempest viktor_nokia created the page https://wiki.opnfv.org/tempest_brahmaputra_page 08:29:44 jose_lausuch: this is indeed confusing since the number of tests differs in ~80...200 range 08:29:55 morgan_orange: I am opening at the requirements.pip, If have some question,I will ask you later 08:30:04 May-meimei: ok 08:30:49 and apex/onos => not started due to lib issue, apex/odl => 15 passed (3 failed) 08:31:01 I was writing the same:) 08:31:12 on compass last night it was 18/18 passed so the bug we used to have in Arno seems to be solved 08:31:19 nice 08:31:35 how about success criteria for test suites? 08:31:41 amaged__: raghavendrachari pbandzi could you have a look and help us to troubleshoot current odl suite on the different installers 08:31:53 viktor_nokia: in theory or in practice... 08:32:03 :) 08:32:05 hi morgan_orange 08:32:08 both :) 08:32:12 ok 08:32:59 "In theory, theory and practice are the same. In practice, they are not." Einstein 08:33:10 I mean how we define status in those brahmaputra testing pages? 08:33:20 morgan_orange: lol 08:33:43 Initially I would have liked to use the dashboard and avoid wiki pages that are unmaintanable.. 08:34:12 for tempest we define our own list 08:34:22 so we may expect a 0 failed 08:34:25 morgan_orange: hi morgan,onos do not support vPing,how to modify this yaml? 08:34:33 test-dependencies: 08:34:33 doctor: 08:34:33 installer: '(apex)|(fuel)' 08:34:33 functest: 08:34:33 vims: 08:34:33 vping: 08:34:34 tempest: 08:34:35 rally: 08:34:35 odl: 08:34:36 scenario: 'odl' 08:34:36 onos: 08:34:36 scenario: 'onos' 08:34:50 but it is impossible, the introduction of odl for instance triggers lots of network errors 08:35:21 so the criteria is OK we we have a reasonable number of tests failes (10%) assuming that we can explain the reason why it fails... 08:35:47 OK 08:36:18 morgan_orange: BTW on apex/onos Zhanghaoyu May-meimei lixiaoguang => ImportError: No module named pxssh ----i'm fixing it 08:36:30 Zhanghaoyu: the proper way to exclude vPing from onos is to add a constraint on scenario so something like scenario: nosdn|odl|ocl 08:36:42 should exclude any scenario with onos 08:36:53 May be we can use colors (green for 100% yellow for 90-100% and red for others) 08:36:56 but vPing is very basic 08:37:08 could you detail offline the reason why it should be excluded? 08:37:56 viktor_nokia: the difficulty is the complexity of combinations...I think it will be hard to reach 90% on all the scenario...but we can try 08:38:14 for vPing is basic => ok or failed 08:38:36 for vIMS => we do not check the results of the tests but the capability to deploy the orchestrator, deploy the vIMS and run the tests 08:38:57 for odl => as compass succeeded to get a 18/18 it should be the target for any odl scenario... 08:39:06 morgan_orange: good news: we got 18 tests total, 18 passed, 0 failed in Fuel+ODL ! 08:39:11 for onos ups to ONOSFW to decide its criteria 08:39:24 for tempest => 90% OK 08:39:26 https://build.opnfv.org/ci/view/functest/job/functest-fuel-opnfv-jump-2-daily-brahmaputra/22/consoleFull 08:39:30 for rally => 90% OK 08:39:39 and error failures must be documented 08:39:44 is is ok for everybody? 08:39:59 sure 08:40:02 OK 08:40:20 morgan_orange, jose_lausuch: so we don't have to pass all the testcase? 08:40:30 but for tempest we have missleading result numbers... we have to figure it out 08:40:50 I will check them 08:40:51 May-meimei: I think if we require that constraint, release B won't happen :D 08:40:54 #info success criteria: vPing 100% OK, odl 100% ok, Tempest 90% OK, Rally 90% OK, vIMS 100%OK (deploy+run tests do not care of test result) 08:41:30 we move to the compass installer? 08:41:33 morgan_orange: CNlucius will explain the reason why onos not support vPing testcase 08:42:10 onos does not support metadata yet, so vm can't get userdata 08:42:12 morgan_orange, jose_lausuch: sorry I left for a while, about compass+onos, vping test is fail, I think jose_lausuch understand the reason, yes? 08:42:14 Zhanghaoyu: ok thanks, if needed we may make some adaptations (as spawning 2 VM in a private network and performing a ping between tehm seems a reaonable testcase....) 08:42:16 Zhanghaoyu: CNlucius is it user data problem ? 08:42:37 #info status compass installer 08:42:51 yes 08:43:07 #info issue vPing towards compass/onos 08:43:08 CNlucius: we can think of another way of doing that maybe 08:43:23 instead of passing userdata, we pass a script and SCP to the vm to ping the other vm 08:43:27 is that ok? 08:43:35 I think it makes sens to adapt vPing if needed ot keep the testcase as it is a basic Hellow world 08:43:49 and we get rid of user data... 08:43:53 morgan_orange: can we force config_drive? 08:43:57 the problem is 08:44:07 we need floating IPs if we want to SCP 08:44:21 so cloud-init can get userdata from config_drive? 08:44:24 carey_xu: then, are we sure config_drive will work always? 08:44:37 That would be best 08:45:08 OK let's discuss that offline 08:45:08 jose_lausuch: need verify 08:45:12 there is 15 minutes left 08:45:22 I would like to speak about rally and documentations 08:45:30 carey_xu: CNlucius: let's discuss it after the meeting 08:45:32 and feature projects.. 08:45:49 just info the lats main results for the installer 08:46:15 #info status fuel 08:46:23 #info on track 08:46:28 #info status joid 08:46:53 #info still lots of errors, but David_Orange told me that ODL has been upgraded yesterday... 08:47:07 vPing is also in timeout on joid 08:47:27 odl fully failed on joid/odl 08:47:37 (only installer with this status) 08:47:43 it could be because of a different port 08:47:49 #info vPing is also in timeout on joid 08:47:55 #info odl fully failed on joid/odl 08:48:01 maybe its not 8181 08:48:17 jose_lausuch: yep but we asked several times the confirmation of the 8181 port, maybe an @ issue 08:48:21 troubleshooting needed 08:48:32 #topic rally 08:48:46 #info rally-cert scenario ready 08:49:13 #info they may need some config adaptations but it is possible now to switch from our traditional way of running rally to this mode 08:49:32 #info are we ok to move assuming that we did not perform manual tests (not access on all the labs) 08:49:41 #info number of tests almost the same 08:50:22 juhak: please review this 08:50:22 https://gerrit.opnfv.org/gerrit/#/c/6897/ 08:50:29 I would suggest to migrate ..then troubleshoot on the different installers 08:50:29 to avoid printing all the output 08:50:51 jose_lausuch: what do you think? 08:50:55 I tried it manually and it works, it prints only the stdout 08:51:05 I think it would be good to do some manual testing with rally-cert first 08:51:32 morgan_orange: +1 with rally cert, but also +1 to test first manually 08:51:38 I can do that on fuel lf-pod2 08:51:40 on which lab can we do that => LF POD1 (apex) and LF POD2 (fuel) 08:51:50 I have a time window for testing 08:51:50 May-meimei: could you try a manual run on compass using this script 08:52:36 #info test rally-cert on LF POD1, LF POD2, Huawei-US and Intel POD5 =>synchro with juhak, jose_lausuch, morgan_orange if OK migration to rally-cert 08:52:49 target test today... 08:52:56 ok? 08:53:04 ok 08:53:07 ok 08:53:19 looks like a busy day 08:53:21 ok! 08:53:30 #topic doc 08:53:36 * jose_lausuch likes challenges 08:53:40 #info developer doc initiated (under review) 08:53:53 #info user and installation/config doc to be patched 08:54:21 question on onos, do you have your own doc, or do you want to be integrated in these documents (we host odl test doc) 08:54:50 if you want to be described in the doc, you must patch the current documents there are sections for ONOS.. 08:55:19 #action morgan_orange May-meimei jose_lausuch test rally-cert and do a synch with juhak for scenario migration in CI 08:56:00 #action lixiaoguang May-meimei update user guide to include onos doc (if not own onos documentation / otherwise just indicate a pointer) 08:56:08 ok 08:56:15 any question on the documents that are supposed to be freeze also today... 08:56:16 morgan_orange: ok 08:56:38 #action carey_xu CNlucius check userdata problem and propose a solution to adapt vPing to make it work 08:56:47 also me 08:57:22 there is also a B plan in progress on test dashboard, the demo last thursday was a bit disapointing and clearly not up to date 08:57:25 jose_lausuch: OK 08:57:35 ok 08:57:37 morgan_orange: thank you so much, and we are handling docs 08:57:40 morgan_orange: what is plan b? 08:57:43 we are considering using our data results (from the DB) through an existing framework 08:58:40 pbandzi: frankbrockners have deployed a ELK in a VM where I moved the mongo DB, I am able to consume the APi to generate through a cron job data ready for graphs 08:58:52 I have to see ghow to integrate them in graphana or .. 08:59:05 ah ok, that one looked very good 08:59:07 it is not critical compared to the rest of our activities 08:59:23 I think if we consider that dashboard looks more "pro", we should use that one 08:59:30 morgan_orange: im new to this meeting, so whats the schedule of this meeting in recently, i want to join this. 08:59:33 but as the test projects made the effort to centralize their results and even provide post processing algo, it will be a pity not to use them 09:00:29 Zhanghaoyu: the meeting is the test group weekly meeting, time is not convenient for Asia (Thursday 3PM UTC) but we have also a slot for Asia (Wedesnday 8 AM UTC) 09:00:54 unfortunately last time nobody attended so I sent a mail to ask whether we should stop this slot 09:01:11 this meeting was created to share information betwwen all the test projects 09:01:30 ok we are one minute late 09:01:35 thanks for joining 09:01:41 good luck for this new very long day 09:01:42 morgan_orange: you are a good guy~ this meeting helps me alot 09:01:42 thanks morgan_orange 09:01:51 everyone has something to do :) 09:02:02 thank you so much 09:02:10 Zhanghaoyu: Jose wrote it in the Functest in depth presentation...we are good guys :) 09:02:21 we are nice guys :) 09:02:25 if you have question keep on this chan 09:02:41 :) 09:02:42 and we are looking for musicians to jam during the summit.. 09:02:48 #endmeeting