08:00:13 #startmeeting Functest weekly meeting May 16th 2017 08:00:13 Meeting started Tue May 16 08:00:13 2017 UTC. The chair is jose_lausuch. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:00:13 Useful Commands: #action #agreed #help #info #idea #link #topic. 08:00:13 The meeting name has been set to 'functest_weekly_meeting_may_16th_2017' 08:00:16 Hi everyone 08:00:46 Hi 08:00:51 #topic role call 08:00:56 #info Cédric 08:00:58 #info Morgan 08:01:02 #info Jose Lausuch 08:01:16 #chair morgan_orange 08:01:16 Current chairs: jose_lausuch morgan_orange 08:01:38 #info agenda for today: https://wiki.opnfv.org/display/functest/Functest+Meeting#FunctestMeeting-16/05(8UTC) 08:01:43 #info Shuya(OOL) 08:02:13 #info Juha Kosonen 08:02:20 #info SerenaFeng 08:03:36 #topic Action point follow-up 08:03:46 not many APs 08:03:46 #info AP: HelenYao sync with jack Chan for internal API 08:04:01 she is not here 08:04:20 #action HelenYao sync with jack Chan for internal API 08:04:24 #info AP: https://www.surveymonkey.com/r/7ZNPMMR 08:04:27 #info survey closed. 08:04:36 but we have new awards 08:04:45 feel free to nominate 08:04:47 #info AP Anual Awards: https://wiki.opnfv.org/display/DEV/2017+Annual+Award+Nominations 08:05:33 #topic Decission on onos and onos_sfc 08:05:35 #info Linda Wang 08:05:38 #info Some comments in JIRA: https://jira.opnfv.org/browse/FUNCTEST-820 08:05:55 LindaWang: I still don't see any onos-sfc scenario in CI 08:05:58 what's the plan? 08:06:19 Yes, they are still under discussion. 08:06:53 to decide if it will be in CI or not 08:06:54 ? 08:07:03 It depends the their work process and whether can make it or not. 08:07:15 It depends on the their work process and whether can make it or not. 08:07:53 on master I am not sure we should systematically assume the heritage of previosu version for features. it could make sense to disable all the feature tests, up to feature project to enable it if they plan to support it in next version 08:08:16 what do you suggest then? 08:08:27 I think there are no major changes from feature projects 08:10:02 LindaWang: can you follow up and keep us posted about the decission? 08:10:10 excluding all the feature project tests for a new release => meaning asking explicitely feature projets to re enable it would allow to know who will be the guy discussing with him 08:10:21 jose_lausuch: Sure. 08:10:23 #info LindaWang want will follow up with Huawei team to see if onos_sfc will be part of Euphrates or not 08:10:47 we have MS2 08:10:57 I hope David will declare that in a table or something 08:11:00 to have visibility 08:11:09 and to know what is the test plan 08:11:26 I know of many feature projects that still want to test what they are testing :) 08:11:30 using this mechanisms will be more explicit, we need a patch to enable the suite...so it is somehow more official than a name in a wiki table... 08:11:49 what are the plans for having that in the DB? 08:11:57 we need to work on that 08:12:14 the info we have in testcases.yaml 08:12:59 we could ask for teh DB declaration as part of the patch to enable the suite in testcases.yaml 08:13:36 but then, what do you suggest, disable all the tests now? 08:13:44 all the feature tests 08:14:06 mmmmm 08:14:08 no sure about that 08:14:10 if no change, reabling will be quick 08:14:21 and we can have a check list: people, DB, .. 08:14:26 maybe we can do it of those we have doubts about 08:14:42 I know they guys for SFC, they still want the test in CI 08:14:45 same for BGPVPN 08:14:49 I guesss same for Doctor 08:15:13 No they are only defined in testcases.yaml. It's quite simple to comment the section. 08:16:05 there are some guys that are checking constantly CI to see if the tests pass or fail 08:16:07 Barometer as well 08:16:08 or set ci_loop to never 08:16:16 not sure if it's a good idea to disable all at once :) 08:16:39 or add flag in testcase section to disable. 08:16:42 ollivier: that's better actualyl, better than a lot of comment lines 08:16:43 it is easier to explain...same for everybody 08:16:54 if you are poart of the release, just reenable it through a patch 08:17:03 ollivier: that's even better! 08:17:15 morgan_orange: ok, we can do that 08:17:15 potentially no test in Master for some hours.. 08:17:18 if everyone agrees 08:17:29 but we need to communicate that properly 08:17:32 But I'm rewriting run_tests.py. Please wait before adding this simple flag. 08:17:34 otherwise, we will get a lot of complains 08:17:50 I thought about the "enabled" flag for a long time already 08:17:53 it is a way to automate MS2 08:17:53 it's a good idea 08:18:20 I can also tell David to communicate that to the PTLs. Another step (besides the DB) is to enable it in testcases. yaml 08:18:26 it is better than uncommenting... 08:18:47 for me the declaration of test cases in DB shall be part of this patch 08:18:54 We should forbid creating a process to call a python script too. 08:19:11 the declaration of the DB is done though swagger 08:19:28 sure part means done at the same time.. 08:19:31 yes 08:19:35 MS2 criteria 08:19:36 ok 08:19:40 let's do that then 08:19:51 we then need to wait for the "enabled" flag from cedric? 08:19:58 or shall I try to work on it? 08:20:31 if he started rewriting run_test... 08:20:44 I would prefer rewriting run_tests and its related UT before. For 5/06 too? 08:20:45 ollivier: when do you think it will be ready? 08:21:04 test_run_tests.py os partially false and takes time 08:21:04 when is MS2? 08:21:10 I don't like to put that much work on you man :D 08:21:43 22th this month 08:21:51 next week 08:22:01 I can work on that, it shouldn't be too difficult 08:22:06 ollivier: and you can focus on your current patch 08:22:13 I'll try to do it today for review 08:22:35 if you like but the next patch will be huge 08:23:08 but rebasing a flag management will be quite simple 08:23:10 ollivier: ok no problem. It's been more than 1 week that I don't look at code due to Summits and travels, I need to do something useful now :) 08:23:32 I think we jumped to the next topic on framework :) 08:23:36 yes 08:23:41 #topic Progress on framework 08:23:41 #topic Progress on framework 08:23:44 #undo 08:23:44 Removing item from minutes: 08:23:51 hehe 08:23:57 we didn't info anything yet 08:23:58 let me do it 08:24:02 #info lots of changes provided by Cedric 08:24:18 #info run_tests also under refactoring 08:24:20 #info Testcases.yaml to provide a flag "enabled" for each test case 08:24:43 #agree automate MS2 for the feature test integration using this future flag 08:24:53 #info the plan is to disable all the feature tests and the MS2 criteria will be that the feature PTLs should enable if they plan to have the tests in CI 08:25:06 #action jose_lausuch work on the 'enabled' flag patcvh 08:25:24 #info each feature project will have to propose a patch to (re) enable its test suite 08:25:38 #info along with the declaration in the DB (if not done yet) 08:26:29 maybe ollivier can give a status on the framework evolution, lots of changes since some weeks...and gate still not broken :) 08:27:10 I think testcase and feature are finished. 08:27:27 We must know focus on vnf and to complete our Framework 08:27:36 #info src ollivier refactoring of testcase and feature OK 08:27:49 #info still work on vnf (sic) 08:28:13 Please review https://gerrit.opnfv.org/gerrit/#/c/28545/ to add UT 08:28:27 #action morgan_orange valentin review vnf accoring to framework changes 08:28:52 my plan is today is to review all the pending patches that I haven't had the time yet 08:28:56 I got a request from Steven 08:29:00 we should merge asap the vping 08:29:13 but we are closed to finish. We have fully cover vnf, add docstrings then to check pylint. 08:29:21 thanks ollivier 08:29:25 great job 08:29:35 #info create Jenkins job for api doc 08:29:38 the more I read your code, the more I feel miserable... :) 08:29:42 in python 08:29:54 morgan_orange: me too :D 08:29:54 that's false! 08:30:12 thumpUP 08:30:31 ollivier Great job 08:30:43 Thank you all! 08:30:49 yes, great job 08:30:53 good 08:30:55 so 08:31:01 what about offline testing 08:31:09 I will take that with Helen offline 08:31:18 there is a patch about providing images to the container 08:31:21 need to dig into it 08:31:22 #info to publish framework prez' 08:32:18 on my side I meanly worked on the reporting page and collaborative part of it 08:32:19 #action to publish framework prez' 08:32:26 the catalog field is now available in the DB 08:32:34 I need to update Functest case with this new field 08:32:47 #action morgan_orange update test case description with new catalog description field 08:33:00 and I need to work on VNF part.. 08:33:06 cool 08:33:20 We need also to see what we do for scenario promotion 08:33:35 yes 08:33:39 at the end of the CI when publishing teh results we should also push the scenario results into teh DB 08:33:44 Can another core review the next changes https://gerrit.opnfv.org/gerrit/#/q/status:open+project:functest+branch:master+topic:use_pretty_table ? 08:33:46 and maybe set the test trust indicator 08:33:47 you invented a cool algorithm to promote tests as well 08:34:22 ollivier: ok, I will 08:34:27 it is a relative trivial invention... 08:34:30 BTW not connected 08:34:37 to share with functest members 08:34:44 yes? 08:34:47 there is a discussion on the testing working group on the infra resources 08:34:50 and the related testapi APIs for update scenario is not very good, I find out a better one, I will try to implement them as soon as better 08:34:59 it seems there is a consensus to say that we will have 2 types of activities 08:35:07 validating version N, qualifying version N-1 08:35:18 morgan_orange: when is that being applied? 08:35:49 #action to create a Jenkins job for api doc 08:35:57 based on previous release, it is not reasonable to believe we can perform load/stress/robustness tests => we need a stable version, impossible to do it on master due to late integration 08:36:06 so we need the 2 chains master and stable for the tests 08:36:20 discussions initiated by mail, topic slot for the weekly meeting this week 08:36:30 the idea is to present that to infra group & TSC 08:37:04 it means also that some feature integration could also be done on stable until candidate release is ready... 08:37:12 which weekly? testing wg? 08:37:15 yes 08:37:22 ok 08:37:27 it is early discussion, but it may have some impacts... 08:37:27 I can attend this week 08:37:37 master is more important than stable. 08:37:53 ollivier: for the release yes 08:38:22 for a Service Provider it is important to have a qualificaiton (load/robustness tests) towards your system... 08:38:25 both are important 08:38:26 if master is stable, you can limit stable testing :) 08:38:33 but clearly Master must be alsways available 08:38:33 release testing... 08:38:47 master is continously development 08:39:00 it should be stable but it's not always the case (although doing a good job) 08:39:01 and installers integrate lots of upstream.. 08:39:19 but for stress testing we should rely on a released "stable" version 08:39:27 not only for Functest 08:39:36 as morgan says, installers stable as well 08:39:39 they have also not a full control. For Danube we were supposed to have stable candidate beginning of January, it was available beginning of March, and the release date did not change 08:40:36 it was for instance impossible to perform a 1 week test to check stability. Even functest we run it once after a fresh installation. Running it N times during a week would be useful to detect stability issues 08:41:13 anyway it isanother topic :) 08:41:15 ok 08:41:17 yep 08:41:24 let's see how the discussion develops 08:41:31 but we really need to fix the resources 08:41:51 another thing 08:42:02 regarding the discussion last week about Dovetail cherry picking 08:42:13 what about my proposal of creating a "stable" tag on master? 08:42:36 even though if we cherry pick all the stuff to stable/danube as ollivier has done , it doesn't solve their problem 08:42:48 they need features that will be part of Euphrates 08:42:53 like offline testing 08:43:16 Are you sure that stable will mean stable? 08:43:32 if they use danube.3.0 won't be enough anyway 08:43:43 it's not the best solution 08:43:47 but... 08:43:54 we should stop backporting features. Normally we should only fix bugs even if they were too late before the release. 08:44:06 I agree 08:44:11 features are for master 08:44:13 stable for fixes 08:44:21 then they must use master. That's the game? 08:44:28 that's my proposal 08:44:32 but not latest master 08:44:42 a tagged version that we can decide at some point 08:45:07 a tag for an alpha euphrates 08:45:14 something like that 08:45:20 we can decide the name later 08:45:32 I would prefer keeping master as stable as possible than creating tags on master (does gerrit allow that?) 08:45:48 ok so we are all in line... 08:45:48 I think so 08:45:53 just need to find the name of the tag 08:46:13 unstable or alph or dovetail will be better than stable? 08:46:19 stable mays be a bit misleading... 08:46:23 #info create a tag in MASTER branch when Functest meets the Dovetail requirements 08:46:30 #info tag name to be decided 08:46:41 unstable is not maybe good :D 08:46:43 We can create release candidates but will they be managed by the project PTL or by the release manager (see ODL)? 08:46:56 ollivier: that's up to us 08:47:03 Debian unstable is much stable that Ubuntu LTS :) 08:47:19 Release manager only worries about final release 1.0 2.0 3.0 and so on 08:47:32 oh really? 08:47:41 I was thinking unstable from a Debian perspective... 08:47:41 but the name is not very appealing 08:47:47 I see now 08:47:49 we will decide later 08:47:51 not now 08:47:58 but yes, let's agree on that 08:47:58 so 08:48:05 #topic Status on Kumar VNF's catalogue 08:48:11 shall we then abandon the cherry picks in stalbe/danube now? 08:48:11 Kumar are you around? 08:48:14 1 sec 08:48:16 this is important 08:48:17 We could check how ODL community handles all release candidates. 08:48:17 #undo 08:48:17 Removing item from minutes: 08:48:23 ollivier: ? 08:48:51 it is done and ready..so we can do it and adopt the rule to never do it again... 08:49:18 is it merged? 08:50:30 It should have been. 08:50:35 ok 08:50:38 we keep it then 08:50:43 too late 08:50:49 sorry for not reviewing last week 08:50:50 #topic Status on Kumar VNF's catalogue 08:50:52 https://gerrit.opnfv.org/gerrit/#/q/status:open+project:functest+branch:stable/danube+topic:backport_refstack 08:51:18 no, it's not merged then 08:51:21 shall we then abandon? 08:51:21 Kumar are you around 08:51:25 ofc not. 08:51:32 ollivier: let's take it after the meeting 08:51:35 let's talk about VNF catalog 08:51:54 Kumar... 08:52:02 #info Kumar made a demo during the plugfest 08:52:07 he is not here 08:52:08 #he also created a wiki page 08:52:17 #link https://wiki.opnfv.org/display/DEV/VNF+Catalogue 08:52:45 nice wiki 08:52:45 #info he will be in Beijing for the Summit 08:52:57 #info LF suggest to extend the internship 1 month 08:53:16 #info his work shoudl be included in sample_vnf project 08:53:23 #info the link for the demo 08:53:34 #link http://ec2-34-210-8-241.us-west-2.compute.amazonaws.com:3000/ 08:53:47 #action morgan_orange move from amazon hosting to opnfv resources 08:53:55 the icons should be clicable 08:54:28 5 minutes left 08:54:32 #topic New internship proposals 08:54:39 thanks for the update 08:54:39 #link https://wiki.opnfv.org/display/DEV/2017+Intern+Projects 08:54:41 #info new proposals created on our admin pages 08:54:49 can you list the new proposals? 08:55:02 #link https://wiki.opnfv.org/display/functest/Functest+admin 08:55:56 ok 08:55:57 #info proposals: vCDN, vPing evolution, VNF/Juju, Automation of performance test post processing, integration of InSpec (security) , integration of upstream K8 test suites, 08:56:01 are they in the intern project page? 08:56:11 vPing it is done with snaps.. 08:56:25 do we need an intern for that? I think it's covered by snaps 08:56:32 VNF/juju it has been done, I asked Narinder to provide the feedback to see if we can include it in Functest 08:56:47 yes these 2 topics are no more relevant (old proposals) 08:56:48 can we add TestAPI to the internship? 08:57:04 there are so many works now 08:57:07 we can add whatever we want... :) just need to have mentor 08:57:11 web portal/authentication 08:57:16 for the moment the proposals on admin page are just proposals.. 08:57:36 when we agree (mentor/topic) we will move them to the intern portal 08:57:41 ok 08:57:42 agreed 08:57:46 SerenaFeng: go ahead :) 08:57:50 yep 08:58:12 thank you. 08:58:14 k8 topic means we have some community lab with kubernetes, only 1 joid scenario today...it is maybe too early 08:58:26 Inspec was suggested by Luke during the plugfest 08:58:44 I have access to a cengn pod with joid 08:58:46 not sure what he did on openSCAP will ne neither maintain or extended 08:58:56 I'd like to write a test case for containers 08:59:04 jose_lausuch: so could be ok all the more as it was more or less for cengn student (starting internship now) 08:59:26 ok so we can contact Dave in cengn and suggets to do it with him 08:59:29 who? in which project? 09:00:28 Dave Urschatz 09:00:35 ya, I know Dave 09:00:38 I mean the intern 09:00:41 we are out of time 09:00:46 and we missed AoB 09:00:51 #topic AoB 09:00:53 anyone? 09:00:59 #action morgan_orange contact Dave (Urschatz) to see if container suite could be done with functest/cengn mentor 09:01:17 currently swagger website of testapi is broken 09:01:29 some bugs introduced by my recent patches 09:01:44 SerenaFeng: Can't read swagger JSON from http://testresults.opnfv.org/swagger/resources.json 09:01:49 I didn't check it with nginx, i am trying to fix it 09:01:51 yes 09:01:55 ok, no problem 09:01:57 that problem solve 09:02:01 another come up 09:02:05 sorry about that 09:02:12 I'm wearing a Nginx t-shirt today :) 09:02:14 so it should wokr! 09:02:17 I think push results still work, only swagger 09:02:23 shall we create some Jiras for Euphrates? 09:02:38 ollivier had some suggestions about the way we use JIRA 09:02:43 yeah, that make sense 09:02:44 we need to discuss it 09:02:47 just to be sure we are not missing anything from our plan drawn during the plugfest 09:02:52 but yes, we need to tidy up JIRA for Euphrates 09:02:53 at least for the new feature 09:02:59 that's a PTL thing, so don't worry too much 09:03:00 my job 09:03:01 :) 09:03:08 ok need to drop 09:03:12 ok 09:03:14 have a good week 09:03:14 #endmeeting