08:01:05 #startmeeting Functest weekly meeting october 10th 08:01:05 Meeting started Tue Oct 11 08:01:05 2016 UTC. The chair is morgan_orange. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:01:05 Useful Commands: #action #agreed #help #info #idea #link #topic. 08:01:05 The meeting name has been set to 'functest_weekly_meeting_october_10th' 08:01:09 #topic call role 08:01:14 #info Morgan Richomme 08:01:23 #info Helen Yao 08:01:24 #info Juha Kosonen 08:01:35 #info Jose Lausuch 08:02:02 #info agenda for Today: https://wiki.opnfv.org/display/functest/Functest+Meeting 08:02:05 #info SerenaFeng 08:02:21 #info hideyasu_ool 08:02:22 #info meimei 08:03:22 #topic action point follow-up 08:03:35 #link http://ircbot.wl.linuxfoundation.org/meetings/opnfv-functest/2016/opnfv-functest.2016-10-04-07.59.html 08:03:50 #info AP1: jose_lausuch morgan_orange plan finalization of Colorado postmortem during OpenStack Summit 08:03:58 #info no update, meeting planed during the OpenStack Summit Wednesday 9-12h 08:04:04 #action jose_lausuch morgan_orange plan finalization of Colorado postmortem during OpenStack Summit 08:04:10 #action AP2: morgan_orange jose_lausuch initiate presentation with proposals for the - and -- alreday referenced 08:04:15 #info see AP1... 08:04:25 #action morgan_orange jose_lausuch initiate presentation with proposals for the - and -- already referenced 08:04:32 #info AP3: morgan_orange refactor Danube page to reflect the introduction of a scenario table in the API 08:04:35 still need to work on that 08:04:38 #info done 08:04:43 #link https://wiki.opnfv.org/display/functest/Functest+Danube 08:04:51 #info AP4: morgan_orange ask feature projects already integrated in Functest for feedback on a possible evolution considering Feature project as an object 08:04:56 #info mail sent, no feedback received so far 08:05:03 #info AP5: all review CI evolution 08:05:09 #info we will see in the next section 08:05:26 any question on the action points? 08:05:52 nope 08:06:00 OK let's move to next topic then 08:06:01 no 08:06:12 #topic PTL election 08:06:25 #info let's formalize the vote for our new PTL 08:07:32 #info I received mandate from Zhanghaoyu, lixiaoguang, Lanqinglong and Valentin 08:07:32 did you create the gerrit commit? 08:07:59 no I will launch a bvote than action me to modify the INFO, I thinkw we can do it through the IRC vote 08:08:09 ok 08:08:16 is everyone here? 08:08:20 commiters? 08:08:41 yes committers 08:08:42 :) 08:08:50 so if you have the +2 -2 power 08:08:50 here 08:08:54 #startvote: Vote for Jose as PTL? Yes, No 08:08:54 Begin voting on: : Vote for Jose as PTL? Valid vote options are Yes, No. 08:08:54 Vote using '#vote OPTION'. Only your last vote counts. 08:09:02 #vote yes 08:09:10 #vote Yes 08:09:12 #vote Yes 08:09:13 #vote yes 08:09:23 #vote yes 08:09:23 #vote yes 08:09:27 #vote yes (proxy Valentin) 08:09:27 morgan_orange: yes (proxy Valentin) is not a valid option. Valid options are Yes, No. 08:09:33 #vote yes 08:09:37 # vote Yes 08:09:41 my 3 proxy votes 08:10:01 :) 08:10:18 I think we have the quorum, only Cedric has not voted, he told me he voted yes but i do not have mails to prove it :) 08:10:22 #endvote 08:10:22 Voted on ": Vote for Jose as PTL?" Results are 08:10:22 Yes (5): viktor_t, SerenaFeng, juhak, morgan_orange, May-meimei 08:10:32 ok proxy is not working 08:10:41 but anyway congratulations to José! 08:10:52 congratulations 08:10:53 congratulations! jose_lausuch 08:10:56 Thanks everyone!! 08:10:59 congrats Jose! 08:11:01 #action morgan_orange modify Info in repo + inform TSC 08:11:03 @jose_lausuch congrats 08:11:03 HelenYao: Error: "jose_lausuch" is not a valid command. 08:11:12 #jose_lausuch congrats 08:11:15 I hope to do it as good as Morgan is doing :) 08:11:20 jose congratulations 08:11:25 jose_lausuch: congrats 08:11:27 thanks :) 08:11:35 no concern, you will be great as you are already great 08:11:41 jose_lausuch: congrats 08:11:55 #chair jose_lausuch 08:11:55 Current chairs: jose_lausuch morgan_orange 08:11:58 thanks ! 08:12:20 I will run the meetings as of next week then 08:12:26 yep 08:12:32 I finish this one :) 08:12:36 If you are OK... 08:12:52 #topic Colorado post mortem 08:12:52 I'm happy to be PTL and having Morgan as support, I think we will do great for Danube 08:13:14 thanks for your votes 08:13:30 #info still time to report your concerns/criticisms/remarks...as said in action points, Jose and myself will plan some formal presentation to tackle the - and -- points 08:13:41 #link https://etherpad.opnfv.org/p/Colorado-Testing-postmortem 08:14:09 yes, let's work on that during this/next week 08:14:27 #topic CI Evolution / adaptation in Functest 08:14:42 this morning I add a figure in https://wiki.opnfv.org/display/functest/Functest+Danube 08:14:54 on CI evolution: Latest, Daily, Weekly 08:15:06 it is linked to the proposal made by Infra working group on CI evolution 08:15:11 https://wiki.opnfv.org/display/INF/CI+Evolution 08:15:20 I would like to include in this section the docker changes proposed 08:15:29 ok 08:15:31 like having different docker containers 08:15:36 this will also impact ci 08:15:41 and our framework 08:15:56 Is unit testing part of CI/CD evolution? 08:16:11 we must include unit testing 08:16:28 I would not say it is part of CI/CD evolution, it should have been done already :) 08:16:29 ollivier: for functest at leastr 08:16:42 but its more a functest thing than ci 08:16:43 but we must include unit test run on each commit before merging 08:16:58 we 'll use unit test in gerrit gates 08:17:01 It will be great to run unittests for every commit... 08:17:03 yes 08:17:06 we have an internship proposal on the topic, and start getting candidates (planned to be discussed in next section) 08:17:06 yep 08:17:15 agree 08:17:20 but definitively in Danube, each commit will trigger unit tests... 08:17:30 for every thing we implement in functest there should be a unit test 08:17:48 the internship will cover unit test on existing utility classes 08:17:57 #info discussion on unit testing 08:17:58 that's something for the internship 08:18:11 #info in Danube Functest unit tests must be run on each commit to stabilize Functest code 08:18:28 #info new dev must be provided with associated unit tests 08:18:35 for openstack_utils for example, we should be able to write test that use each function 08:18:52 we can probably quickly test it, i can actin myself for that 08:19:11 the question is 08:19:19 if there is a change in file X 08:19:30 should we run only tests for that file ? 08:19:38 or we run a generic bunch of tests? 08:19:42 no unit tests are short...re run them all 08:19:46 same for all commits? 08:19:50 ok 08:20:09 but there are a lot of functions 08:20:28 do we need a deployment for that? 08:20:34 but for the moment there is no test... 08:20:45 ya, sure :) 08:20:54 we speak about unit test with mock object 08:21:00 we do not need deployment 08:21:05 ok 08:21:13 virtual deployements must be used by installer to run smoke tests 08:21:17 it is another type of tests 08:21:18 there was a presentation from mark beierl about that 08:21:20 good idea 08:21:45 yes maybe we could initiate a wiki page....there are lots of testing, but clearly unit testing was missing... 08:21:50 we are growing... 08:21:52 #action morgan_orange see how to trigger unit testing on commit on master 08:21:56 Could unit tests be run in functest containers? 08:22:07 ollivier: why not? 08:22:08 yes 08:22:31 Serena for the testapi includes unit tests that were runnable inside the conainer 08:22:36 So I must update https://gerrit.opnfv.org/gerrit/#/c/22921/ 08:23:03 python-mock is not listed in Dockerfile 08:23:13 ok 08:23:20 we can include it easily 08:23:24 we will discuss this proposal in next topic 08:23:25 in requirements .pp 08:23:30 no 08:23:43 back to CI evolution 08:23:49 to be sure that we are sharing the same view 08:23:51 It should be installed via deb as python. 08:23:52 include it in test-requirements.txt 08:23:56 no 08:24:07 please don't use pip if not required 08:24:07 apt-get? 08:24:13 sure 08:24:22 in infra they want to create latest (test case not trustable), daily (4 iterations OK) and weekly (was Ok so run the scenario less often) 08:24:34 ollivier: can you propose a patch in the dockerfile? 08:24:47 we had in Colorado a different view: daily = short & mandatory, weekly = long 08:24:49 as soon as I switch to my public network 08:24:53 so at the end we never run weekly... 08:24:55 the commit is ready 08:25:04 ollivier: ok 08:25:06 morgan_orange: yes 08:25:16 and now we have 1 more stage 08:25:18 I think we can converge to Infra view and realocate TrustIndicator to latest/daily/weekly 08:25:34 that is the idea of the proposal made on the wiki 08:25:41 weekly doesnt mean necessarily every week 08:25:47 it includes also the creation of a scenario object in the test result databae 08:26:54 jose_lausuch: you mean not every week for a given scenario 08:27:17 I mean, its another step in the trust indicator 08:27:30 how do we say something can go to weekly? 08:27:42 CI evolution taks about promotions 08:27:48 my view was to add a promotion stage at our level 08:27:56 if latest works, then it becomes daily 08:28:09 yes, if 4 daily works it becomes weekly 08:28:14 when do we say daily is good enough to be considered weekly? 08:28:24 exactly 08:28:31 but when do we run weekly? 08:28:33 weekends? 08:28:34 but we should take case....scenario versus cases 08:28:43 yes it was planned to run on the week end 08:28:47 or after 4 succesful dailies? 08:29:37 ok 08:29:56 and when will weekly return to dailyF? 08:30:17 except for when we run weekly it fails 08:30:20 if weekly fails then it will be promote to daily (in the CI indicator) 08:30:36 we should also include if the testcase is changed, right? 08:30:55 if the testcase is changed/new it shall be initiated as latest 08:31:03 if the test case changes, then we need to start from scratch... 08:31:07 yes 08:31:29 but we do not face dramatic change in a test case, do we? 08:32:08 usually, but we should take exception into account 08:32:12 it means also that we have some changes in our test processing: preparation => tests => scenario as shown in the figure 08:32:17 SerenaFeng: sure 08:32:40 any objection to use Trust Indicator for tagging the case latest/daily/weekly? 08:32:54 not at all 08:33:01 I agree with that approach 08:33:15 totally agree 08:33:16 (initially we planned a 0 to 1 indicator + history) here it will be more basic but for promotion from daily to weekly we will need the history, 08:33:17 but I'm affraid we'll never reach weekly... 08:33:21 as it has happened 08:33:31 hmm 08:34:00 weekly is our huge blueprint, I guess 08:34:19 we will have weekly scenarios by construction...and in most of the case we will have test case to run on it 08:34:47 if you consider http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-compass.html 08:35:24 2 scenarios are 100% OK and candidate for weekly 08:35:34 that is also what I want to point 08:35:49 ok 08:35:52 a scenario can be promote from Daily to Weekly if all the tests are Daily 08:35:56 great 08:35:59 whatever is ready for weekly we run it 08:36:03 and a scenario owner may descide to overwritte the default list 08:36:29 #info morgan_orange suggests a new mechanism for test and scenario promotion in line with CI evolution 08:36:49 #info changes to be planned - reallocation trustIndicator + new APi methods on scenario 08:36:57 #action SerenaFeng jose_lausuch review proposal 08:37:21 ok jose_lausuch you want to speak about the docker evolution discussed with HelenYao? 08:37:36 yes, we can do it now 08:37:47 how many topics are left? 08:37:53 #topic evolution of docker 08:38:15 internship + global class 08:38:21 5 minutes.. 08:38:23 ok 08:38:24 so 08:38:34 we can use different dockers 08:38:43 for excample 1 docker for vims 08:38:51 another docker for feature projects 08:39:13 for example, promise tests need a couple of libraries that are installed just for the test case 08:39:14 #info our docker is becoming fat 08:39:17 all the other feature projects using one docker? 08:39:23 one docker per feature is also feasible 08:39:27 that's the question 08:39:33 what do you think? 08:39:39 #info we need to create several dockers not just one (e.g. vIMS, feature,..) 08:39:42 I agree with HelenYao 08:40:02 I think make one docker small enough to implement one complete process 08:40:04 there are many features 08:40:09 is that scalable? 08:40:32 it depends on how we design it 08:40:37 someone needs to maintain all the docker images in dockerhub 08:40:44 #info discussion 1 docker per feature versus 1 docker for all feature tests 08:40:45 I'm doing that currently 08:41:00 so 08:41:17 #info question on the scalability and the maintenance/versioning of all the docker if we go for 1 docker / feature tests 08:41:18 Isn't there a automatic way to do it? 08:41:19 https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml 08:41:38 I see 11 featuret ests 08:41:39 tests 08:41:51 I remember fatih ask me if I want to automatically dockerize testapi 08:41:56 we can envision more features in the future 08:41:57 #info currently 11 feature projects declared in Functest config 08:42:13 #info and it may only grow... 08:42:15 shall we identify which tests need more stuff to be installed in the docker image? 08:42:20 and isolate them? 08:42:24 packaing all features into one image can make the image very large 08:42:30 yes 08:42:49 but some feature test dont need extra packages 08:42:54 then we need to separate them once again 08:43:27 ok we can see the issue, we will not solve here, cna I action HelenYao SerenaFeng and jose_lausuch to study this offline? 08:43:39 #action jose_lausuch identify which packages are instaleed in the docker build process and for what reason 08:43:40 sounds good 08:43:49 yes please 08:43:55 ok 08:43:55 interesting discussion :) 08:44:00 #action HelenYao SerenaFeng jose_lausuch study impact on a massive dockerization of feature 08:44:08 we should be allowed to build the container locally too. 08:44:09 I think for VNF it is pretty clear 08:44:12 do we have any JIRA item for the docker stuff 08:44:14 the only issue is for feature 08:44:29 #action morgan_orange create a JIRA_epic on docker topic 08:44:36 #topic Global class 08:44:42 and we must must take care of the layered filesystems... 08:44:44 I would like also to discuss cedric's proposal 08:45:01 #link https://gerrit.opnfv.org/gerrit/#/c/22921/6 08:45:29 +1 to this proposal 08:45:30 #link https://jira.opnfv.org/browse/FUNCTEST-353 08:45:50 ollivier: still need time to review the code, hopefully today/tomorrow 08:45:56 I sent a maiol to feature project, so I did not get feedback whteher this is a complexification or simplification for them 08:46:13 +1 to the proposal 08:46:20 hwoever, i think it is mandatory to adop a more structured way for our discussions with feature project 08:46:30 +1 08:46:31 morgan_orange: its a simplification for everyone I think 08:46:38 jose_lausuch: no pb. I can also publish the related ODL part if you quite agree 08:46:42 I agree ... but... 08:46:48 just we and they need to learn a new way to integrate tests in functest 08:46:55 ollivier: makes sense to have an application 08:47:07 with odl 08:47:41 morgan_orange: it's already done. But I think it shouldn't be part of this review. 08:47:46 ollivier: sure, go ahead 08:47:51 #info everybody seems happy to create some organization into the chaos of featur test integration 08:47:54 ok 08:48:13 then we will have to adapt the other test projects 08:48:19 morgan_orange: I will split into two reviews but please wait odl part ot merge both 08:48:36 morgan_orange: before merging both 08:48:37 OK 08:49:07 for the moment we put the exit code and exceptions in the file, we had a discussion offline on where we should put it 08:49:15 the unit testing class is at the same level 08:49:35 maybe we should create a test directory to put all our future unit tests and map the current directories 08:50:10 yes 08:50:13 it think it is better than putting the test file with the file, but I am not a python senior expert... 08:50:13 morgan_orange: exit codes must be outside (in releng). It just allows working without releng. 08:50:19 a dir with unit tests makes semse 08:50:41 Ok we can do comment in the review 08:50:49 but great job 08:50:57 #topic internship 08:51:13 if you remember we put some proposals on line 08:51:38 #link https://wiki.opnfv.org/display/DEV/Intern-projects-page 08:51:48 it seems that it is Internship time in India 08:51:57 we got 2 proposals for unit testing 08:52:11 #info 2 candidates for https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Functest+unit+tests 08:52:25 call with one planned on Thursday 08:53:02 jose_lausuch: shall we plan another slot with the second students, or could we invite and detial other possible internship... 08:53:26 morgan_orange: we could setup a second slot 08:53:41 we could create one on security group, as it seems that both have an ODL background, it would be great 08:53:57 yes, good idea 08:53:58 but 08:53:58 jose_lausuch: OK I contact him to see when we can have a skype/hangout/.. 08:54:09 we need to talk to Ray first 08:54:25 I'll ask him 08:54:50 what will you ask him? 08:55:13 if we can have 2 interns 08:55:25 if there is budget for that 08:55:37 it is a questions of mentors I think but OK you are right, let's see with him 08:55:43 #topic AoB 08:55:53 any topic, questions you want to raised? 08:55:59 i have something 08:56:07 some of you already know maybe 08:56:15 I created a module in releng 08:56:16 #info Barcelona Functest meeting planned on Wednesday 9-13h 08:56:47 #info disucssion on adapater done in releng 08:56:49 #link https://gerrit.opnfv.org/gerrit/#/c/22599/ 08:57:02 #info Installer Adapters is a library to get information about the deployment and get the ips of the nodes, ssh, scp, etc https://gerrit.opnfv.org/gerrit/#/c/22599/ 08:57:14 ]feel free to add you and review 08:57:31 I will keep example.py and will do unit tests in further commits 08:57:35 for the review, I satrted seing alias 08:57:41 please add sheebang 08:57:48 what about the copyright? 08:57:48 I wonder if we can ask one for functest to avoid adding them one by one... 08:57:53 morgan_orange: alias? 08:58:01 functest-reviers in gerrit 08:58:14 ollivier: copyright? 08:58:37 morgan_orange: that would be great 08:58:45 what's the policy about copyright in OPNFV? 08:58:59 #action morgan_orange ask for functest gerrit alias 08:59:33 https://wiki.opnfv.org/display/DEV/Contribution+Guidelines 08:59:37 ollivier: ya, right, i just copy pasted from other files 08:59:40 I will ask 08:59:45 seems OK 08:59:53 Copyright (c) 2015 and others. 09:00:08 i could copyright it to myself 09:00:23 but 2015? 09:00:25 until you do not overwritte widely copyright from others.... 09:00:29 should it be 2016? 09:00:32 I don't like listing the authors 09:01:03 jose_lausuch: yep 2016 09:01:07 morgan_orange: should I add Orange to the testcases base? 09:01:32 ollivier: i think you can 09:01:42 not sure you will have additional money... :) but you can 09:01:46 haha 09:02:10 one question from me: any background information about bitergia? I hope to know it before attending the coming sync meeting 09:02:23 basically it is a little bit redundant with git 09:02:32 apache2 must be mentioned for their automatic scanning 09:02:39 yes HelenYao go ahead 09:03:09 any link that I can refer to about bitergia? what is the agenda for the sync meeting with them? 09:03:43 HelenYao: we met them very recently, they just made a presentation in berlin then more recently during the weekly meeting 09:03:55 The agenda is under construction 09:04:18 basically they are subcontractor for OPNFV and provides the traditionnal dashboard for git 09:04:22 they addeed the jenkisn stuff 09:04:24 morgan_orange: jose_lausuch: OpenSTack removes all authors from file several years ago and I think the Fundation own the copyright 09:04:28 and our discussion was on the testing 09:05:00 see 09:05:17 ollivier: we can raise the question to the TSC, I think in the CLA we signed we also transfer the copyright to Linux Foundation 09:05:27 hi I'd like to discuss https://gerrit.opnfv.org/gerrit/#/c/22487/, it has been there for a few day 09:05:29 #action morgan_orange check copyright stuff with Ray 09:06:23 I mean to make the code more clear, without changing any logic it used to be 09:06:23 morgan_orange: It will be simple if all python files are homogeneous 09:07:42 so should I keep it? 09:07:45 regarding SerenaFeng's patch I saw the exchanges, I am a bit lost, I see that it is initiailly a reformatting patch (so very limited risk even if lots of files are impacted) 09:08:42 I have tested vping rally tempest parser, I don't have resources to test the others 09:08:59 I don't see the prupose if we load the config file for every attribute 09:09:20 #action jose_lausuch: ollivier morgan_orange SerenaFeng final decision on patch https://gerrit.opnfv.org/gerrit/#/c/22487/ 09:09:35 we load the config file for every attribute even if we using the original one 09:10:13 let's finalize it offline 09:10:16 True. But if you define a new class, please save the values instead of only defining getters. 09:10:30 since we will separate docker later, it is not necessary we load it all 09:10:42 because every testcase will be a new start 09:11:03 maybe we can continue after I close the meeting (so stay on the chan...) 09:11:10 ok 09:11:22 thanks everybody for your participation and your confidence for the 3 first OPNFV release 09:11:27 big thanks to Morgan for the excellent work done as PTL 09:11:32 I was very happy to PTLize this project 09:11:49