08:00:02 #startmeeting Functest weekly meeting May 17th 08:00:02 Meeting started Tue May 17 08:00:02 2016 UTC. The chair is morgan_orange. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:00:02 Useful Commands: #action #agreed #help #info #idea #link #topic. 08:00:02 The meeting name has been set to 'functest_weekly_meeting_may_17th' 08:00:12 #info Jose Lauushc 08:00:12 #topic call roll 08:00:17 #info Morgan Richomme 08:00:19 #info Viktor Tikkanen 08:00:20 #info Juha Kosonen 08:00:22 #info SerenaFeng 08:00:33 #info CG_Nokia (ColumGaynor) 08:00:39 #info agenda: https://wiki.opnfv.org/display/functest/Functest+Meeting 08:00:52 #topic Action Point Follow up 08:01:00 #link http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2016/opnfv-testperf.2016-05-10-08.00.html 08:01:16 #info Action Point #1: viktor_nokia jose_lausuch troubleshoot healthcheck, instances don't get an IP on Nokia's POD with ODL 08:01:21 #info May 08:01:59 #info healthcheck problems were investigated together with Apex experts 08:02:15 it seems to be an OLD thingy 08:02:16 #info lhinds 08:02:21 #info seems that flow creation takes too long time (minutes) 08:02:32 email from Jamo? 08:03:03 Jamo referenced some old bugs but this is probably different one 08:03:24 ok 08:04:09 #info Action Point #2: Viktor_nokia remove duplicated/obsolete files in the repo 08:04:25 done 08:04:29 #info done 08:04:50 #topic Plugfest summary 08:05:07 #info Plugfest took place last week in Colorado (Cable Labs) 08:05:37 #info main participants: Huawei, Intel, China Mobile, Windriver, Canonical, Redhat, EMC, Arm, Linux Foundation, Orange 08:05:46 #info plugfest: functest run successfully towards the different systems 08:05:47 great success right ? 08:05:51 #info tests done over 2 local clusters Intel & Huawei 08:05:56 #info several config tested 08:06:02 #info on huawei: fuel-onos, compass-nosdn 08:06:07 it was nice helping until late hours in europe :p 08:06:08 #info on Intel: mirantis fuel-nosdn, windriver, apex-nosdn 08:06:16 #info tests also done on remote cengn joid/nosdn lab 08:06:21 #info results stored in plugfest DB 08:06:29 #info some logs shared 08:06:36 #info globally adaptation of Brahamputra functest for the plugfest was simple 08:06:45 #info thanks to José and David late support 08:07:01 can you share the results url? 08:07:04 #info Functest, yardstick and storperf were introduced to the participants 08:07:23 #link http://plugfest.opnfv.org/testapi/results 08:07:35 results are anonymzed (except cengn) 08:08:02 test successfully run on Apex but not stored in DB and no time to relaunch them befor leaving 08:08:15 nice 08:08:17 but everything in place on Intel local POD (a bay they brought to cable labs) 08:08:29 was there any orange box? 08:08:35 note that tests were done on a pure fuel/mirantis and on windriver 08:08:39 with NUCs? 08:08:43 no orange box 08:09:08 only 1 guy from Ericsson the first day 08:09:14 but nobody in the the labs 08:09:35 note that the most ambitious testcase was the one managed by China Mobile / Windriver / Intel / Redhat 08:10:03 they performed an interconnection between 2 Clouds for a vEPC scenario, which was quiete nice 08:10:18 wait for the official summary to get details 08:10:39 #info next plugfest planned in France beginning of December for Colorado 08:10:59 #info we can see 4 types of activities during such plugfest (at least in #1) 08:11:07 #info knowledge sharing on testing 08:11:11 #info VNF testing 08:11:16 #info install party 08:11:16 who was there from Ericsson? 08:11:26 #info interco/interop testing 08:12:34 jose_lausuch: hmmm do not remember but I will try to find back 08:12:38 no problem 08:12:49 any question on this first plugfest? 08:13:02 nope, glad to hear good news and success! 08:13:30 what is interco/interop testing? 08:13:52 test a VNFs distributed in 2 clouds for instance 08:14:03 so Apex versus windriver 08:14:12 for instance with VNFs distributed in both clouds 08:14:33 ok, thank you 08:14:44 both installed on a hardware people discover the same day 08:15:03 initially the ambitious would have been to also run all the installer in a given scenario (odl_l2) on the same hardware... 08:15:12 but we did not have time to do it 08:15:30 but it would also have been interesting (to compare apple with apple..) 08:15:32 next time 08:15:47 and next time we may imagine more VNFs and interop 08:15:56 it is a good neutral place to trey such things 08:16:24 #topic Sprint #7 follow-up 08:16:40 #info we are almost at the end of Sprint 7 (Sprint 8 to be started this week) 08:16:48 #info refactoring completed 08:16:57 #info CI run back to nominal 08:17:20 #info Test API swagger integration also almost ready 08:17:33 good progress so far 08:17:39 #info CLI completed 08:18:01 SerenaFeng: if you want to complete, I just tested before taking the plane, looks nice, verion was fine, some issue with other methods but the GUI fully OK 08:18:04 as regards to testAPI swagger 08:18:29 it needs lots of modify to the original code 08:18:40 without any test, I dare not to do it 08:18:48 so, I am working on the unittest 08:18:57 so far, the framework is almost done 08:19:01 can you show us the GUI life? 08:19:06 live* 08:19:07 I will make the git review today 08:20:36 then I want to write the unittest first, with the unittest's guarantee, I will continue with the swagger work 08:20:37 jose_lausuch: I take a picture and share it 08:20:38 is that OK? 08:20:44 SerenaFeng: yes 08:21:17 BTW during the plugfest I discussed with Mark Beier from Storperf, they have (as Yardstick) unit tests and nice coverage indication without Jenkins 08:21:25 place for improvement for us in D release 08:21:25 ok 08:21:57 For the current Jiras, any concerns? 08:22:06 and I found that, we don't have pep8 check(use tox) in our testAPI project 08:22:25 I think it is necessary 08:22:38 so do we need pep8 check? 08:23:12 I think it is better to add the pep8 check 08:23:16 we have pep8 check in Functest 08:23:22 it is already activated 08:23:27 in Releng, not sure 08:23:41 we may ask fdegir 08:24:02 #action morgan_orange ask fdegir if we could add pep8 check on releng (useful for the test API) 08:24:03 ok 08:24:06 #link http://postimg.org/image/hk7eleakx/20683b3d/ 08:24:11 its not in other repos 08:24:15 just functest 08:24:28 SerenaFeng: anyway, you can install flake8 locally 08:24:31 and to a local scan 08:24:38 I do that before commiting anything :) 08:24:48 the violations you get on gerrit are not enough 08:24:53 I posted the image of the swagger interface in the link 08:24:59 ok, thank you 08:25:00 you get only 1 message of each violation type 08:25:04 better to do it locally 08:25:16 I have a question on the reporting 08:25:18 just pip install flake8 08:25:24 and then run it against your directory 08:25:26 flake8 ./ 08:25:40 now that we decided that scenario will be validated by Healthcheck+smoke + scenario specific tests 08:25:45 I don't do the pep8 before, I will do it later, thank you 08:26:05 I planned to refactor a little bit the current reporting -includign the call to the new test case yaml description file) 08:26:18 jose_lausuch: will we push results for the health check? 08:26:41 I assume no, but if healthcheck fails then we do not execute the other tests.. 08:26:44 shall we? 08:26:50 yes 08:26:51 well 08:26:57 its a very simplet test 08:26:59 it doesnt say much 08:27:02 just verify the API 08:27:06 agree 08:27:07 but sometimes it fails (Nokia lab) 08:27:22 for the smoke we have the tests in pink in https://wiki.opnfv.org/display/SWREL/Test+Release+Criteria 08:27:27 but for our verification automatic reporting page it owuld be niace to have 08:27:32 regarding healthcheck: it's not a very simple test :) 08:27:42 the test itself is simple 08:27:52 it covers the whole neutron+ODL chain 08:27:55 if it fails, its not so simple to understand why :D 08:28:14 well, yes, but the tests just boots 4 Instances :) 08:28:24 if that doesnt work, we are in trouble! 08:28:36 vping_ssh/userdata are more simple :) 08:29:24 mmmm 08:29:29 nope 08:29:40 vping is about pining instances 08:29:47 healthcheck doesn't do any pining 08:29:53 it's the step before 08:30:01 just bringning up instances 08:30:08 healthcheck create a lot of objects (users/projects/images/volumes/networks/subnets/floating IPs/routers/router interfaces etc. 08:30:40 healthcheck for vm ? 08:30:42 and assumes that e.g. ODL is functional in SDN scenarios 08:30:43 the only thing vping_ssh does not create is volumes 08:31:44 healthcheck is not supposed to be SDN or feature dependant 08:31:58 vping also creates /images/networks/subnets/floating IPs/routers/router interfaces etc. 08:32:10 it corresponds to basic commands at the end of an OpenStack installation 08:32:13 the only difference is that it creates only 2 VMs 08:32:16 healthcheck 4 08:32:29 ok so not only basic commands 08:32:30 agree 08:33:28 so, lots of duplicated test scenarios in healthcheck and vping? 08:33:37 why do we need both of them? 08:33:39 It is OK for me; I just afraid that in some cases we will not get other test results if healthcheck will fail 08:33:49 quite time-consuming 08:33:57 if healthcheck fails, it doesnt make to run other things :) 08:34:19 why not? 08:34:19 how can you sell something to a telco/operator where you can't create instances normally? 08:35:03 SerenaFeng: the first tests are very short, ~ 1 or 2 minutes max 08:35:05 SerenaFeng: not really, less than 5 minutes in total :) 08:35:51 For example, if flow creation is delayed 2 minutes (as in Apex setup 08:35:57 Ok let's see in practice in the next weeks if it is OK to deal with a healthcheck first (if ok => run smoke tests else tests not triggered) 08:36:01 healthchek will fail 08:37:10 viktor_nokia: do you think if we increase the timeout it will pass? 08:37:24 we can do that 08:37:30 It passed when I added 1 min delay 08:37:36 we can do that 08:37:43 that's fine 08:37:46 as long as it works in the end 08:37:46 but is it acceptable to have delay in flow creation? 08:38:09 should we create an APEX ticket? 08:38:13 timeout=60 currently 08:38:25 can you push a patch with the value you have tested and it works? 08:38:32 I think it makes sens to report it to Apex 08:38:42 I can do it 08:38:53 but honestly... more than 1 minute to boot a cirros VM and get an IP... is not telco grade :D 08:38:58 viktor_nokia: ok, thanks 08:39:11 jose_lausuch: I agree :) 08:39:28 according to jenkins it is OK with fuel and joid 08:39:33 on Master 08:39:47 withouth timer changes 08:40:01 viktor_nokia: let's do that then, let's add more time 08:40:03 3 min max? 08:40:37 I'm not sure, it will delay the whole test 08:40:52 because we need to wait before launchng instances 08:40:59 #action viktor_nokia increase timeout value for healthcheck 08:41:15 #action viktor_nokia create Jira on Apex regarding the timeout issue in healthcheck test 08:41:23 not really reasonable 08:41:54 Let's see with Apex team first 08:42:08 One possibility is to not require IP addresses during healthcheck 08:42:11 wait before launching? why? 08:42:14 as our main goal is to have a better control of the time, if we start bu increasing the timeout on test #1... :) 08:42:42 jose_lausuch: because cirros tries only three times 08:42:43 the idea I had about waiting for the IP is that if that fails, vping_tests can fail as well (if they dont get ips) 08:42:51 viktor_nokia: ah ok 08:42:54 mmmm 08:43:12 viktor_nokia: ok, let's add a delay before creating the instances then 08:43:14 but in apex case vping passes since it creates less networks :) 08:43:16 let's make it generic 08:44:30 another point on the smoke test cases 08:45:02 we have still the security_groups test mentioned 08:45:18 I think it will be hard for David to do it (baby in approach...) 08:45:52 ollivier: or anyone volunteer to implement this testcase or shall we report it to Colorado SR2 or D river? 08:46:16 ohh nice 08:46:21 baby coming 08:46:27 #info question on security_groups tests 08:46:30 I don't want to volunteer... too many things already :) 08:46:40 if its not possible, then for D 08:46:57 #info resource issue on David side, 2 options => someone volunteer to replace David, or move it ro D release 08:47:08 are they neutron security groups or something else? 08:47:16 morgan_orange, i can take a look at it 08:47:29 viktor_nokia: yes 08:47:33 let me find the link 08:47:33 main thing is I want to get this scap scanning finished first 08:47:44 #link https://gist.github.com/davidblaisonneau-orange/bb74fb1a9d4a87552fc2cdf68608f69f 08:47:51 #action lhinds sync with David for secrurity_group test cases 08:47:55 good transition 08:47:56 this is the test description 08:48:00 lhinds: upadte on security scan 08:48:14 we did not mention in the Test Criteria 08:48:20 https://wiki.opnfv.org/display/SWREL/Test+Release+Criteria 08:48:27 but for me it should be in blue 08:49:05 FYI: tempest_full contains number of scenario test cases for security groups 08:49:17 security scanning is almost there now. I found a way to make the SSH hop in paramiko, but I need to get a better view of how we can get SSH keys from each installer, and how do we discover the IP addresses in functest? 08:49:36 Second question you all should be able to help with 08:49:49 viktor_nokia: good point, I think David had a look, let's sync if testcase is needed or already covered by Tempest full 08:50:20 If its blackbox, do you have discovery of IP addresses (endpoint URL perhaps) or do you enter this manually? 08:50:28 trying to work out what hooks I can use 08:52:24 we got info from CI (INSTALLER_IP) and retrieve IPs also from scripts releng/fetch_os.sh The approach is different from one installer to another 08:53:18 for the ODL test case for instance we need th IP of keystone, Neutron and ODL 08:53:23 we do it through keystone_ip=$(openstack catalog show identity |grep publicURL| cut -f3 -d"/" | cut -f1 -d":") 08:53:23 neutron_ip=$(openstack catalog show network | grep publicURL | cut -f3 -d"/" | cut -f1 -d":") 08:53:23 odl_ip=${neutron_ip} 08:54:26 we can discuss it offline (based on the mail you just sent) 08:54:42 would I be able to get the IP of other nodes in a similar way..I need compute / control and network 08:55:01 ok..if you can reply on where I can read more, that would be useful 08:55:32 #action morgan_orange jose_lausuch answer lhinds mail (how to retrieve dynamically IP for security test) 08:55:46 #action morgan_orange add security test in test Criteria page 08:55:50 put me as CC 08:55:53 it will be challenging :) 08:56:12 maybe we can do it for opnfv installers 08:56:13 viktor_nokia: OK I think everybody is in CC 08:56:31 but not for others, and ollivier will stand up :) 08:56:34 and otherwise, I have ported all argparse values into the the config_functest.yaml 08:56:56 shall we plan a demo next week? with a GTM? 08:57:16 morgan_orange, let me see what shape I am in at the end of the week 08:57:20 demo? 08:57:29 if you meant secscanning that is 08:57:35 yep 08:57:41 demo of the scscanning 08:57:53 how long does it take on your POD? 08:58:20 let's see at the end of the week, if ready we may plan a lived demo next tuesday 08:58:26 I can show it working outside of functest, but need to get it integrated before I show it running in functests docker context 08:58:28 and probably a demo of SerenaFeng'as APi 08:58:36 lhinds: ok 08:58:46 we are a bt late 08:58:53 #topic Berlin Summit 08:59:07 #info progams has been announced 08:59:26 #info several presentation to be planned (for the Design and the Summit) 08:59:53 #action morgan_orange check the Functest related pres and see with involved people 09:00:04 who will be in Berlin? 09:00:11 we planed to make the testAPI session, I will make the proposal later 09:00:12 I will 09:00:20 me 09:00:24 me 09:00:24 Ich auch 09:00:28 me too 09:00:32 I will 09:00:34 :) 09:00:41 ok most of us, cool 09:01:11 #info Functesters in Berlin: lhinds, jose_lausuch SerenaFeng raghavendrachari viktor_nokia ollivier morgan_orange ... 09:01:16 #topic AoB 09:01:25 anything you want to share before we close the meeting 09:02:11 I have a question, I have a Jira, where I planned to add tempest errors on the tempest reporting 09:02:31 #link http://testresults.opnfv.org/reporting/index-tempest-apex.html 09:02:47 is it OK to add in the details section when we push teh results this list 09:03:19 I think it is the easiest way, otherwise I have to get the full logs on the artifcats and match the id betwwen the different runs 09:03:39 ok 09:03:40 yep 09:03:44 I think it is simple to add the testcases failed directly in teh results in addition of teh nb tests run / failed / .. 09:04:10 #info question on the opportunity to put the Tempest failed case in the results pushed to the DB 09:04:31 ok if any objection, I planned to do it thsi way 09:04:35 any other topic? 09:04:41 no objections 09:04:41 yes 09:04:45 anual awards opened 09:04:49 vote for morgan_orange! 09:05:02 no no vote for jose_lausuch! 09:05:10 both :) 09:05:13 +1 +1 09:05:15 haha 09:05:20 +1 +1 09:05:26 :p 09:05:50 morgan_orange: do you think, it's a better idea to make the testAPI a real project, I mean we can make the installation of it? 09:05:51 morgan_orange is the plugfester 09:06:17 last info, it seems that installers will be late... (fuel already mentioned that mitaka nosdn will not be available immediately...) 09:06:43 SerenaFeng: what do you mean by we can make the installation of it? 09:06:55 something independent from releng? 09:07:16 contains setup.py in it 09:07:37 just like nova do 09:07:47 yes we could do that 09:07:53 for the moment it is very manual 09:08:00 yeah 09:08:02 I clone the repos on the testresults.opnfv.org a 09:08:19 and I run the python result_test_api ... 09:08:24 we have to download the project, run it as it is 09:08:24 we could have something better 09:08:41 you got the meaning, thank you 09:08:50 idem for the test case we could also imagine for D or later rivers to use this API as a baseline for all the test projetcs to run the tests 09:09:03 I don't know how to express myself, 55 09:09:36 yeah 09:10:05 ok enjoy the ned of Sprint 7, see you next week for Sprint 8 09:10:07 #endmeeting