08:00:27 <morgan_orange> #startmeeting Functest weekly meeting January 3rd 08:00:27 <collabot`> Meeting started Tue Jan 3 08:00:27 2017 UTC. The chair is morgan_orange. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:00:27 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic. 08:00:27 <collabot`> The meeting name has been set to 'functest_weekly_meeting_january_3rd' 08:00:33 <morgan_orange> #topic call role 08:00:37 <morgan_orange> #info Morgan Richomme 08:01:11 <ollivier> #info Cédric 08:01:12 <juhak> #info Juha Kosonen 08:01:40 <HelenYao> #info Helen Yao 08:01:50 <morgan_orange> Happy new year everybody! 08:01:56 <morgan_orange> All the best for 2017 08:02:06 <HelenYao> Happy New Year~ 08:02:26 <HelenYao> Did u have a good vacation? 08:02:32 <morgan_orange> too short :) 08:02:56 <morgan_orange> #topic action point follow-up 08:03:10 <morgan_orange> they should be relatively old 08:03:12 <morgan_orange> #link http://ircbot.wl.linuxfoundation.org/meetings/opnfv-functest/2016/opnfv-functest.2016-12-13-08.01.html 08:03:40 <morgan_orange> #info AP1 review https://gerrit.opnfv.org/gerrit/#/c/25725/ 08:04:11 <morgan_orange> I merged it on the 14th but there was a question from Cedric on the 16th 08:04:59 <HelenYao> The problem should not be there, I guess 08:05:25 <morgan_orange> Ok in next section I planned to discuss the status (jenkins/Dev/JIRA) 08:05:50 <ollivier> Please add a comment in this review 08:06:17 <ollivier> if it's fixed (which review?) 08:06:26 <HelenYao> ok 08:06:29 <morgan_orange> #info AP2: morgan_orange ask fdegir if any progress on the capability to jenkins to create a new status (blue/red + orange? balls) => done, it seems it is possible, a specific text must be added in jenkins log. I am pretty sure it was not tested but it should be possible 08:06:50 <morgan_orange> #info AP3 morgan_orange see with Cedric on the improvement of the testcaseBase class => done patch merged 08:07:37 <morgan_orange> #info AP4 morgan_orange see with Jose to suggest a presentation of ABOty during weekly meeting (possible that they are US based..so 8 UTC is a bit hard) => done we suggest a presentation during the Testing weekly meeting (APAC slot as they are based in India) 08:08:19 <morgan_orange> #info AP5 HelenYao morgan_orange juhak Cedric Jose Steve: review this patch (2 scenarios proposed by Serena for config management) 08:08:25 <morgan_orange> SerenaFeng: any update.... 08:08:43 <SerenaFeng> yes, it is already merged 08:08:53 <morgan_orange> #info patch merged 08:08:57 <ollivier> I don't like puting a text to Jenkins... 08:09:00 <SerenaFeng> proposal is selected during last meeting 08:09:19 <SerenaFeng> and I already adapt some modules 08:09:42 <SerenaFeng> but there're some test projects I don't have resources to test them 08:09:42 <morgan_orange> #info AP6 morgan_orange create a wiki page to indicate the story of the logs in functest => done 08:09:44 <morgan_orange> #link https://wiki.opnfv.org/pages/viewpage.action?pageId=8687204 08:10:14 <morgan_orange> ollivier: there is probably a better way, the answer from fatih said that it was possible to have multi status 08:10:19 <SerenaFeng> I hope some people who have resource to do the test can take the job 08:11:14 <morgan_orange> SerenaFeng: which test projects? for which constraints (scenario/installer) 08:11:27 <SerenaFeng> yes, scenario and installer 08:11:38 <ollivier> morgan_orange: so hack run_tests.py. We should keep our internal codes. 08:11:42 <SerenaFeng> I only have fuel with nofeature environments 08:12:50 <morgan_orange> ok, it could be interesting to share on our available config...CI is in a bad shape (see next section) but we are not the only guilty people... 08:13:12 <morgan_orange> any other comments on the old action points? 08:13:53 <morgan_orange> #info jenkins multi status exit => run_tests.py hack to keep our internal codes 08:14:07 <morgan_orange> #topic Status Jenkins 08:14:23 <morgan_orange> #link https://build.opnfv.org/ci/view/functest/ 08:14:44 <morgan_orange> #info Colorado OK but no run since more than 20 days 08:15:27 <morgan_orange> #info so no data in http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html, compass, fuel, joid 08:15:44 <morgan_orange> #info CI is not really continous, isn't it? 08:16:01 <morgan_orange> #info on Danube all jobs are red 08:16:17 <morgan_orange> #info but for different reasons... 08:16:28 <HelenYao> I have been watching the CI during the past days 08:16:42 <HelenYao> most of the problems of Functest have been fixed 08:16:50 <morgan_orange> it seems that there is a pending patch for job refactoring on Danube 08:17:01 <HelenYao> except for rally keystone task on newton 08:17:07 <morgan_orange> #info HelenYao watched CI and most of the Functest related issues are now fixed 08:17:18 <morgan_orange> #info except for rally keystone task on newton 08:17:26 <morgan_orange> but Rally is not blocking 08:17:45 <HelenYao> it breaks rally_sanity 08:17:54 <HelenYao> I am still looking at the issue 08:17:59 <morgan_orange> yep but it does not prevent the other tests to be run 08:18:24 <HelenYao> I have a history record for the build result in the past days. I will send it out later 08:18:28 <morgan_orange> so far regarding the combinations: apex/bare metal => healthcheck issue... apex installation issue? (https://build.opnfv.org/ci/view/functest/job/functest-apex-apex-daily-master-daily-master/lastFailedBuild/console) 08:18:50 <morgan_orange> apex/virtual: not run 08:18:53 <HelenYao> I would say it's installation issue 08:19:08 <HelenYao> the healthcheck calls the openstack CLI 08:20:23 <morgan_orange> but here the issue seems linked to the DHCP (no IP available) 08:20:38 <morgan_orange> most of the API have been sucessfully called 08:21:30 <HelenYao> it's odl l3 scenario and the DHCP issue should be with the odl 08:21:32 <morgan_orange> we got this issue with the joid/LXD scenario. We expect a console feedback, if it has changed or been removed in newton, it could lead to an error 08:21:56 <HelenYao> I talked to the local compass team and for their status, the ODL for newton is not fully supported 08:22:06 <morgan_orange> HelenYao: OK a bit rusty 08:22:43 <morgan_orange> with nosdn scenario healthcheck is OK https://build.opnfv.org/ci/view/functest/job/functest-apex-apex-daily-master-daily-master/364/console 08:23:06 <morgan_orange> we got errors in tempest copper and SNAPS tests 08:23:27 <morgan_orange> #action juhak could you check Tempest issues (as you have a apex env..) 08:23:51 <juhak> morgan_orange: ok 08:23:52 <morgan_orange> ok for apex then 08:23:58 <morgan_orange> compass/bare: issue when cleaning the resources due to OS_IDENTITY_API_VERSION (https://build.opnfv.org/ci/view/functest/job/functest-compass-baremetal-daily-master/lastFailedBuild/console) 08:25:31 <HelenYao> I can take a look at the compass issue 08:26:50 <morgan_orange> compass/virtual: installation issue on huwaei-virtual1 (https://build.opnfv.org/ci/view/functest/job/functest-compass-virtual-suite-master/lastFailedBuild/console) and issue with snapshoting on SNAPS tests with huwaei-virtual4 (https://build.opnfv.org/ci/view/functest/job/functest-compass-virtual-daily-master/lastFailedBuild/console) linked to keystone version 08:27:32 <morgan_orange> #action HelenYao check compass issues (issue with keystone version during snapshot/clean phase) 08:27:39 <morgan_orange> ok for compass 08:27:40 <morgan_orange> for fuel 08:28:14 <morgan_orange> fuel/bare: timeout on vPing... 08:28:24 <morgan_orange> but healthcheck ok (https://build.opnfv.org/ci/view/functest/job/functest-fuel-baremetal-daily-master/lastFailedBuild/console) 08:28:53 <morgan_orange> seems linked to the scenario (ovs) 08:29:24 <morgan_orange> on nosdn-nofeature => same issue than with compass on snapshot in SNAPS (https://build.opnfv.org/ci/view/functest/job/functest-fuel-baremetal-daily-master/1028/console) 08:29:37 <morgan_orange> so fixing this issue should fiw several scenarios 08:30:09 <morgan_orange> fuel/virtual: same issue (https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-daily-master/lastFailedBuild/console) 08:30:48 <HelenYao> One thing to notice, we should take a look at the date of execution. for instance, the https://build.opnfv.org/ci/view/functest/job/functest-compass-virtual-suite-master/ run on Dec.7 08:30:52 <HelenYao> which is very olddated 08:31:28 <morgan_orange> yes, if no run since this date, it is pobably due to the fact that no system was installed or jobs have been canceled 08:32:08 <morgan_orange> but yes forget about huawei-virtual1 - it was just an healthcheck and a long time ago... 08:32:39 <morgan_orange> joid/bare: seems issue in installation (impossible to execute a nova list) - done 6 days ago 08:32:52 <morgan_orange> joid/virtual: never run 08:33:04 <morgan_orange> #action morgan_orange sync with joid to get the status 08:33:22 <morgan_orange> lots of space for improvement in 2017.... :) 08:33:47 <morgan_orange> #topic status Dev 08:34:01 <morgan_orange> #info lots of patches under review... 08:34:14 <morgan_orange> #action all back to patch reviews... 08:34:41 <morgan_orange> for those who were active during the Xmas break, any topic you would like to raise in particular? 08:35:07 <morgan_orange> I saw that the interns (unit tests + testapi) submitted several patches 08:35:25 <morgan_orange> I also saw that Valentin patched my proposal on VNF on boarding 08:35:53 <morgan_orange> jack submitted a proposal for the test landing page and there is a pending patch on scenario methods for the DB 08:36:00 <HelenYao> one topic from me about unit test - we have different opinions about the implementation about unit test 08:36:11 <morgan_orange> so lots of things to review on my side in addition of the 896 mails in my inbox :) 08:36:40 <HelenYao> refer to the comment in https://gerrit.opnfv.org/gerrit/#/c/26143/ 08:37:18 <HelenYao> Comment from Serena -First of all, I let myself in as a reviewer, here's my two cents: 08:37:18 <HelenYao> 1) According to TDD theory, I think unittest should also be E2E, we don't need to write unittest for each method each file, what should do is to simulate sending a cli command and check it is correctly executed. 08:37:18 <HelenYao> 2) what we should test is that a cli command is sent and executed, rather than invoke a method such as 'cli_base.os_snapshot_show' directly. 08:37:52 <HelenYao> Comment from me: About the unit test, I have a different view about it. I think the existing implementation meets the requirement of unit test. Every line of the code should be covered and 100% coverage rate is the ultimate goal. 08:37:52 <HelenYao> For the example you came up with, there are two testcases together to fulfill the same target. The reason why it has two testcases is, the unit test is testing layer by layer. 08:38:06 <HelenYao> Comment from Serena - Helen Yao, I totally agree with you that 100% coverage is needed, but we don't need to achieve it via testing method by method. Unit test should also be function-based. There are some concerns to do it in this way: 08:38:06 <HelenYao> 1) Method is quite unstable, it will be refactored regularly according to new requirements or simply by code refactoring. However functionality is relatively stable and more guaranteeing of the implementation. Moreover by writing unit test functionally, we can do refactor more rest assured. BTW, Continuous refactoring is largely advocated and widespread nowdays. 08:38:06 <HelenYao> 2) There are some properties in Python make the writing of unit test method by method very difficult and quite useless, like monkeypatch, decorator, property? builtin methods in class 08:38:06 <HelenYao> 3) Writing unit test in the unit of function is also what OpenStack did and advocated, and I think they already make a good example and we can follow their step. 08:38:20 <HelenYao> oops, I input too many words here 08:38:29 <morgan_orange> #info HelenYao described the discussion on unit tests, see meeting logs for details 08:39:50 <morgan_orange> usually we try to adopt OpenStack way... 08:39:53 <HelenYao> I agree with Serena that some high level of functional testings are needed but I think unit test should cover the code with every possible case 08:40:11 <SerenaFeng> 2) There are some properties in Python make the writing of unit test method by method very difficult and quite useless, like monkeypatch, decorator, property? builtin methods in class 08:40:25 <morgan_orange> I think ollivier will agree with you 08:40:54 <HelenYao> I would like to see some examples of OpenStack 08:40:57 <SerenaFeng> function-based unit test can also cover every possible cases 08:41:06 <HelenYao> I took a look at openstack and it have the coverage rate gating 08:42:28 <SerenaFeng> for example if we do it method by method, how to test feature project code, since it only has a __init__ function? 08:42:30 <ollivier> morgan_orange: reading... 08:42:37 <morgan_orange> ok interesting, can I ask Ashish SerenaFeng HelenYao to prepare some slides on the different approach for next week? 08:44:04 <morgan_orange> no idea is the topic has been discussed in yardstick/storperf/qtip already 08:44:18 <morgan_orange> #action morgan_orange contact other test projects to get their feedback on unit test startegy 08:44:50 <morgan_orange> #action Ashish prepare a short presentation on unit test strategy (with view on Openstack way) 08:45:18 <morgan_orange> anything else you want to indicate regarding the dev status 08:45:34 <HelenYao> by function-based, I would like to know the granularity. To me, we can have two levels of testings: unit test + functional test 08:45:48 <HelenYao> for unit test, it test the different cases of a method 08:46:10 <HelenYao> for functional test, it can be a scenario 08:46:20 <SerenaFeng> I think it is the old fashion 08:46:32 <SerenaFeng> unit test is function-based, it is not functional test 08:47:00 <ollivier> I don't see the point of functional testing in case of functest. 08:47:34 <SerenaFeng> I think we can let Ashish to OpenStack first, then decide which way to carry on 08:47:52 <morgan_orange> ok I think I understand the debate...maybe it is better to prepare some slides before starting a new Hernani battle :) 08:48:32 <morgan_orange> #topic internships status 08:48:50 <morgan_orange> can you info some news on the progress 08:49:15 <morgan_orange> I think that regarding security_group, I did not see any update (but I have lots of mails to read...) 08:49:23 <morgan_orange> for the VNF catalog, I will sync asap 08:49:32 <morgan_orange> #action morgan_orange sync on VNF catalog 08:49:49 <HelenYao> # Ashish is working on unit tests for cli, ci and utils. The sign-off criteria is at least 85% coverage rate for each module 08:50:03 <morgan_orange> #info Ashish is working on unit tests for cli, ci and utils. The sign-off criteria is at least 85% coverage rate for each module 08:50:24 <HelenYao> i forgot the keyword :-( thank you morgan_orange 08:50:29 <SerenaFeng> #info rohitsakala is working on auto publish TestAPI documents to artifacts 08:50:57 <morgan_orange> HelenYao: as far as i can see, it is not integrated in jenkins now, right? https://build.opnfv.org/ci/view/functest/job/functest-verify-master/2819/cobertura/ 08:51:10 <morgan_orange> on coverage I see the odl and core classes from ollivier 08:51:42 <HelenYao> none of the the unit test patches have been merged 08:51:47 <morgan_orange> SerenaFeng: ok good timing to test it on testresutls.opnfv.org ....as jenkins is very quiet... 08:52:14 <HelenYao> functest/cli.py 0 0 100% 08:52:14 <HelenYao> functest/cli/cli_base.py 66 5 92% 08:52:14 <HelenYao> functest/cli/commands.py 0 0 100% 08:52:14 <HelenYao> functest/cli/commands/cli_env.py 66 55 17% 08:52:14 <HelenYao> functest/cli/commands/cli_os.py 75 12 84% 08:52:14 <HelenYao> functest/cli/commands/cli_testcase.py 36 2 94% 08:52:14 <HelenYao> functest/cli/commands/cli_tier.py 38 1 97% 08:52:15 <HelenYao> functest/core/testcase_base.py 42 0 100% 08:52:15 <HelenYao> functest/opnfv_tests/sdn/odl/odl.py 153 33 78% 08:52:16 <morgan_orange> SerenaFeng: impact will be minimum 08:52:16 <HelenYao> ----------------------------------------------------------- 08:52:16 <HelenYao> TOTAL 476 108 77% 08:52:18 <HelenYao> https://build.opnfv.org/ci/job/functest-verify-master/2818/console 08:52:19 <SerenaFeng> testresults.opnfv.org? 08:52:47 <SerenaFeng> he haven't work on testresults.opnfv.org yet 08:53:11 <morgan_orange> SerenaFeng: the idea is to automate the deployment of the test API + associated doc in testresults.opnfv.org, no? where do you want to publish the doc (only in the container?) 08:53:39 <SerenaFeng> we want to publish the docs in artifacts 08:53:51 <morgan_orange> HelenYao: thanks, could be interesting if the tests are stabilized to include them in the global coverage in jenkins 08:54:47 <SerenaFeng> If we want to publish it testresults.opnfv.org, I think we need to study how to do it 08:54:49 <morgan_orange> SerenaFeng: ok, I believed is was directly linked to the testapi docker and then publishing the container means publishing the doc, that is why I believed the target was testresults.opnfv.org 08:54:59 <SerenaFeng> for now, we working on publishing it to artifacts 08:55:00 <morgan_orange> so we are sure there is no gap between the online APi and the documentation 08:55:20 <morgan_orange> if you publish to artifact you may have a more recent version in the doc than on testresuls 08:55:27 <morgan_orange> we can disucss that offline 08:55:36 <SerenaFeng> ok 08:56:07 <morgan_orange> #action morgan_orange SerenaFeng precise testresults.opnfv/org versus atifacts for testapi doc 08:56:22 <morgan_orange> #action HelenYao ashish consider integration of CLi unit test in coverage for jenkins 08:56:23 <SerenaFeng> I think there are some misunderstanding between us, it is good to discuss it offline 08:56:36 <morgan_orange> #topic AoB 08:56:42 <morgan_orange> 5 minutes left 08:56:54 <morgan_orange> any news from bitergia/Linux foundation? 08:57:10 <SerenaFeng> no 08:57:28 <SerenaFeng> no news from bitergia 08:57:29 <morgan_orange> #action morgan_orange ask for bitergia update to Ray 08:57:57 <morgan_orange> any comment on the vnf onboarding (still I did not read all my mails...) 08:58:27 <morgan_orange> any other topic for today? 08:58:34 <morgan_orange> Jose will be back next week... 08:58:55 <morgan_orange> Danube is planned for end of March...it will come quickly 08:59:00 <morgan_orange> when is Chinese new year? 08:59:09 <SerenaFeng> 27th Juan 08:59:19 <HelenYao> Jan.27, the Spring Festival Eve 08:59:34 <morgan_orange> it is a 1 or 2 weeks break? 08:59:42 <HelenYao> one week break 08:59:47 <SerenaFeng> officially one week 08:59:50 <morgan_orange> you are reasonable... :) 08:59:58 <SerenaFeng> but I think most of people will take two week's leave 09:00:13 <morgan_orange> I understand them...and what about you 09:00:36 <HelenYao> I will not take extra days off, that is, Jan. 27 ~ Feb.2 09:00:38 <SerenaFeng> 1 week and 3 days I guess 09:00:41 <HelenYao> Be back on Feb.3 09:01:01 <morgan_orange> ok thanks, it will be our turn to work :) 09:01:20 <morgan_orange> but Danube will definitively come quickly... 09:01:26 <HelenYao> One bad news, we will still work the weekend of Feb.1 09:01:40 <morgan_orange> thanks for joining today 09:02:02 <morgan_orange> Enjoy the mails/gerrit review for those who are back this week 09:02:20 <morgan_orange> and one more time all the best for 2017! 09:02:24 <morgan_orange> #endmeeting