08:00:22 #startmeeting Functest weekly meeting January 17th 2017 08:00:22 Meeting started Tue Jan 17 08:00:22 2017 UTC. The chair is jose_lausuch. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:00:22 Useful Commands: #action #agreed #help #info #idea #link #topic. 08:00:22 The meeting name has been set to 'functest_weekly_meeting_january_17th_2017' 08:00:32 Good morning/afternoon 08:00:53 #topic role call 08:00:59 #info Jose Lausuch 08:01:17 #chair morgan_orange 08:01:17 Current chairs: jose_lausuch morgan_orange 08:01:19 #info Helen Yao 08:01:29 #info Linda Wang 08:01:44 #info Juha Kosonen 08:01:50 #info Juan Vidal 08:02:07 #info Morgan Richomme 08:02:30 #info rohitsakala 08:03:07 #topic Troubleshooting status 08:03:13 let´s go installer by installer 08:03:19 #info APEX 08:03:24 #info Apex jobs: https://build.opnfv.org/ci/job/apex-daily-master/ 08:03:31 #info Deployments seem stable (blue) 08:03:47 #info healthcheck, vPing, rally_smoke, connection_check and api_check are 100% ok. 08:03:58 #info 1 error in Tempest. Copper and Doctor fail. 08:04:07 I talked to Tim Rozet 08:04:13 and there was a problem with the ODL port 08:04:30 and he proposed this patch https://gerrit.opnfv.org/gerrit/#/c/26983/ 08:05:24 APEX runs = only nosdn-nofeature and odl_l3 08:05:28 anything you want to add about apex? 08:05:33 yep 08:05:46 odl tests are OK in odl scenarios 08:05:52 what is the problem with odl port? 08:06:02 I think they changed it 08:06:06 so now its 8081 08:06:06 ok 08:06:09 not 8181 08:06:16 robot tests were failing 08:06:45 as far as I remember Cedric discussed this config issue long time ago 08:06:54 an installer is using an env variable 08:07:04 for the ip 08:07:05 for others it is default port 08:07:08 but ok 08:07:09 the ports are hardcoded 08:07:37 not always..depends of the installers... 08:07:51 good point for genesis... 08:08:18 I see 9 failures in Tempest 08:08:25 on odl_l3 scenario 08:09:14 do we know the openstack version used by apex 08:09:15 https://git.opnfv.org/functest/tree/functest/opnfv_tests/sdn/odl/odl.py#n150 08:09:15 fuel also uses another port 08:09:15 which is fine 08:09:15 no one is obliged to use a default port 08:09:15 but maybe we could put it in a conf file or something 08:09:15 not in the code 08:09:16 ok 08:09:23 ah really? 08:09:24 ok 08:09:31 I saw yesterday 1 failure in nosdn 08:09:32 ok 08:09:47 #info 9 tempest failures in odl_l3 scenario 08:10:05 i think odl_l3 scenarios have been quite unstable during colorado as well 08:10:22 for odl if the env variables are properly set os.environ['SDN_CONTROLLER_IP'] os.environ['SDN_CONTROLLER'] it is fine... 08:10:41 yep 08:10:48 arg no ok odlwebport is hardcoded 08:10:49 I only see only failure for tempest for odl_l3 https://build.opnfv.org/ci/job/functest-apex-apex-daily-master-daily-master/371/console 08:11:02 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops [202.057694s] ... FAILED 08:11:09 right 08:11:13 ok I had a look at https://build.opnfv.org/ci/job/functest-apex-apex-daily-master-daily-master/369 08:11:15 where did you see 9 morgan_orange_ 08:11:15 ? 08:11:28 maybe the odl fix... 08:11:43 could be 08:12:00 so 1 error to investigate on tempest 08:12:04 + errors on feature projects 08:12:14 #info we think after the odl port fix, tempest have reduced to 1 error only, as in nosdn scenarios 08:12:16 we should maybe contact dopctor and copper to report feedback 08:12:27 and investigate on Tempest 08:12:31 with apex team 08:12:40 #action jose_lausuch contact Copper PTL to see what happens with Copper test case 08:12:51 #action jose_lausuch contact Doctor PTL for Doctor test case 08:13:16 #info all SNAPS smoke tests are OK with apex 08:13:32 juhak: could you have a look at the Tempest error? 08:13:55 morgan_orange: yes that's ok 08:14:15 its always the same: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops 08:14:35 #info tempest error log: http://artifacts.opnfv.org/logs/functest/lf-pod1/2017-01-13_10-55-38/tempest/tempest.log 08:14:43 yep it is not the first time... 08:14:51 nop 08:14:57 I'll take a look 08:14:57 ok, lets move to the next installer 08:15:00 #action juhak troubleshoot on Apex error 08:15:01 #info COMPASS 08:15:12 and we are sure that apex is in newton? 08:15:24 apex is not in newton 08:15:27 I tihkn so 08:15:33 but using keystone v2 I thionk 08:15:35 no? 08:15:41 hehe.. not clear to me 08:15:49 https://wiki.opnfv.org/display/SWREL/Keystone+API+Usage+per+Installer+in+Danube 08:15:52 I will talk to Tim Rozet 08:15:59 we probably should had that at the beginning of the test 08:16:02 in this table, apex will not support v3 08:16:05 v3 doesnt mean newton 08:16:08 ok 08:16:12 keystone v3 is just the version 08:16:17 I tihkn they deploy Newton with v2 08:16:39 #action talk to trozet about openstack version vs keystone version. Can we use v3 as all the others? 08:16:39 exactly. i was wrong 08:16:49 maybe it is not verified and its not in their plans 08:17:22 morgan_orange: to have what? 08:17:22 #info get a snaphsot of teh versions of the main component before doing tests could be useful (for further versions..) 08:18:00 well, not at the beginning but everytime openstack_utils is called, there is a message that says so, but yes, it would be good to have it at the beginning 08:18:10 I think functest is able to run the mainstream version of all component clients 08:18:59 in the functest.log, I cannot see any mention of newton 08:19:01 by version, it can be clientversion, api_version 08:19:11 not newton no, I mean v3 08:19:13 or odl version 08:19:20 we could include that 08:19:30 it's not so difficult 08:19:39 I'm working on it :) 08:19:42 yep my comment is (and it is not for functest only) it would be interesting to know precisely the version of the components of the SUT 08:19:42 what about compass? 08:19:43 the log will prompt about the api_version that is set in the env 08:19:50 yes let's move to compass 08:20:09 2017-01-16 22:00:39,343 - openstack_utils - INFO - OS_IDENTITY_API_VERSION is set in env as '3' 08:20:12 #info deployments are not 100% stable (mixture of red and blue balls) https://build.opnfv.org/ci/job/compass-deploy-baremetal-daily-master/ 08:20:33 #info Rally_sanity fails in keystone scenario. 10 failures in tempest 08:20:37 #info rest of the test cases are ok. 08:20:49 the compass is mainly failing b/c of the rally keystone and request 08:21:05 https://gerrit.opnfv.org/gerrit/#/c/27015/ 08:21:15 I finished the upgrading to new rally 08:21:29 HelenYao: Functest but it does not explain that deployments do not seem stabilized yet 08:21:39 #info filed a fix for keystone problems (also with other installers) https://gerrit.opnfv.org/gerrit/#/c/27015/ 08:21:46 the request failure is blocked by https://gerrit.opnfv.org/gerrit/#/c/27039/ 08:22:36 morgan_orange: sorry, I did not follow ur point 08:22:40 and will the patch affect other installers, as just discussed, v3 is not supported by APEX 08:22:58 the patch is able to support both v2 and v3 08:23:13 HelenYao: Jose mentioned that compass deployments are not always blue. When they are OK, then we have issue in Functest due to Rally as you mentioned. 08:23:25 but they are for the moment not always OK 08:24:51 #action review all https://gerrit.opnfv.org/gerrit/#/c/27015/ 08:24:57 if possible merge today 08:25:02 and see results in Jenkins tomorrow 08:25:22 it is me or gerrit is very slow this morning? 08:25:24 I think the link to compass is wrong. It seems to be https://build.opnfv.org/ci/view/functest/job/functest-compass-baremetal-daily-master/ 08:25:27 HelenYao: why do you clone the latest rally? 08:25:33 all of the latest build failed 08:25:33 we would like to control the versios 08:25:35 versions 08:25:51 the rally is going to release 0.8.0 08:25:57 latest is not a good idea 08:25:58 all current dev is under master 08:26:00 latest can be unstable 08:26:11 the new release is totally different from 0.7.0 08:26:22 we are in the middle of the old release and new features 08:26:26 when is 0.8.0 planned? 08:26:39 some features are incapable of backward compability 08:26:40 HelenYao: https://build.opnfv.org/ci/job/compass-deploy-baremetal-daily-master/ (the deployment) and https://build.opnfv.org/ci/view/functest/job/functest-compass-baremetal-daily-master/ (functest after teh deployment) 08:27:03 0.8.0 will be released in this month. 08:27:16 the keystone v3 is only supported by the new feature 08:27:36 ok 08:27:38 I see 08:27:41 that's the reason why I am using master. once 0.8.0 is released, we will get rally from 0.8.0 08:27:50 but as soon 0.8.0 is out, we should tag it again 08:27:54 sure 08:27:56 ok 08:27:57 ok for me 08:28:22 anything else for compass? 08:28:32 do we know if only apex will not use keystone v3 08:28:38 I am just afaid master will not always work 08:29:02 shall we put rally as unblocking until this is fixed 08:29:10 SerenaFeng: yes, but now stable tag doesnt work either :) 08:29:12 Rally is about to freeze the release 08:29:15 to give a chance to the other tests (even if we know that rally will fail) 08:29:32 well, i think we can give it a shot with latest master 08:29:38 the patch from helen 08:29:40 morgan_orange agree 08:29:45 if we see problems, we block it 08:29:50 yeah, give it a shot in other installers before the rally is released. per my experience, the rally is not that instable 08:30:51 ok 08:31:12 next installer? 08:31:20 the main problem 08:31:40 should we follow the reason why the deployment of campass failed? 08:31:50 sure, go ahead 08:32:46 SerenaFeng? 08:32:50 6 compass deployment OK on the last 17 attempts 08:32:57 but I think it is a compass issue 08:33:05 I think we need to contack compass team to troubleshooting it 08:33:08 then wen deployment is OK, it is up to use to investigate 08:33:22 yes, we can't solve it here :) 08:33:41 #action SerenaFeng get an status update about Compass deployment problems 08:33:48 ok 08:33:49 #info JOID 08:33:58 #info the deployments don´t look stable in CI https://build.opnfv.org/ci/job/joid-deploy-baremetal-daily-master/ 08:33:59 I think it should be better we assign to Helen 08:34:11 ok 08:34:19 #info no joid deployment since 25/12 08:34:21 I will work on it 08:34:25 #action HelenYao get an status update about Compass deployment problems 08:34:35 #info last Functest run was in December 28... 08:34:37 #info joid deployed on Intel pods...Intel pods no more available 08:34:44 that's bad... 08:34:52 #info joid pod available on Huawei but installation not available 08:35:06 #action jose_lausuch talk to jmorgan1 about intel pods status 08:35:29 #action morgan_orange contact narinder to get visibility...would be hard to troubleshoot if no run before the end of january 08:35:38 yes.. 08:35:48 this outage is dangerous 08:35:51 for the release 08:35:55 maybe issue to raise at release meeting 08:36:02 we have milestones 08:36:02 I will 08:36:13 it seem that one cannot be completed here 08:36:32 what if milestones are no met in time (installer integration ready) 08:36:36 question for David 08:36:47 #info fuel 08:36:55 ok 08:36:57 #link https://build.opnfv.org/ci/job/compass-deploy-baremetal-daily-master/ 08:36:59 I have some info here 08:37:03 #info Fuel Deploy jobs: https://build.opnfv.org/ci/job/fuel-deploy-baremetal-daily-master/ 08:37:07 compass? 08:37:09 :) 08:37:12 oops 08:37:14 bad tab 08:37:23 #undo 08:37:23 Removing item from minutes: 08:37:26 #undo 08:37:26 Removing item from minutes: 08:37:29 #info Fuel Deploy jobs: https://build.opnfv.org/ci/job/fuel-deploy-baremetal-daily-master/ 08:37:58 #info despite the red balls in the past days, Functest is executed and surprisingly some tests are ok. 08:38:07 #undo 08:38:07 Removing item from minutes: 08:38:15 irc://irc.freenode.net:6667/#info despite the red balls in the past days in the deploy jobs, Functest is executed and surprisingly some tests are ok. 08:38:17 shit 08:38:21 irc://irc.freenode.net:6667/#info despite the red balls in the past days in the deploy jobs, Functest is executed and surprisingly some tests are ok. 08:38:25 sorry :D 08:38:48 it seems that there is a wrong return value. I can see that it says "deployment is done" at the end of this job for example: https://build.opnfv.org/ci/job/fuel-deploy-baremetal-daily-master/1532/console 08:38:52 anyway 08:39:02 what matters to us 08:39:02 #info Functest jobs: https://build.opnfv.org/ci/view/functest/job/functest-fuel-baremetal-daily-master/ 08:39:15 #info vPing and Tempest are 100% ok. 08:39:28 #info Connection Check test case fails. Keystone in Rally also gives an error. 08:39:40 we may note that 164 tests are run 08:39:51 the basic ops (faile don apex) is nto run on fuelk 08:39:56 I guess rally fails due to the same problem as compass 08:40:05 why not? 08:41:11 the count of tempest test case varies per blacklist 08:41:14 good question....https://git.opnfv.org/functest/tree/functest/opnfv_tests/openstack/tempest/custom_tests/blacklist.txt 08:41:25 it is explicitely excluded 08:41:39 I see Deployment failed in deployment console, will it affect rally? 08:42:11 I hope to see the log of request 08:42:16 oh interesting 08:42:42 we have to check the status of that bug and re-enable that test case 08:42:43 for keystone, I would say the current rally does not support v3 08:43:07 #info rally fails the keystone scenario due to lack of v3 support in 0.7.0 08:43:28 could also expain the errors on rally/request 08:43:33 SerenaFeng: the deployment is completed and succesfful, but I dont know it returns -1 08:43:40 but the jobs exited due to an error on SNAPS 08:43:45 there must be some wrong check at the end 08:43:57 yes 08:43:57 ok 08:44:00 right 08:44:11 keystoneauth1.exceptions.http.NotFound: The resource could not be found. (HTTP 404) 08:44:20 HelenYao: can you contact Steven about this error? 08:44:21 it was reported to Steve but he has no access to a fuel env to test 08:44:26 ok 08:44:27 I have 08:44:29 I can do it 08:44:34 jose_lausuch: I think I already reported the bug to him 08:44:42 I set up the server on friday 08:44:45 and I could try 08:44:46 but he answered he has not the nev to test 08:44:52 ok, thanks 08:45:05 morgan_orange: ok. I will talk to steve 08:45:09 that is why I sent a mail to try to identify which env we have for such issues 08:45:16 #action jose_lausuch test connection_check on a fuel env 08:45:35 thats the next topic 08:45:42 let's move 08:45:42 and it is a real bug - the case could be fail but we should not exit 08:45:54 we have a 5th installer.. 08:45:57 haven't we 08:45:59 yes, we should continue if its not a blocking test 08:46:04 daisy? 08:46:17 yep any news? 08:46:22 #info Daisy 08:46:28 #info no functest runs so far 08:46:38 yes 08:46:47 I provided information to the team, about how to integrate functest jobs, the needed things in fetch_creds and so on 08:46:49 but nothing so far 08:46:52 I think they are working on the integration work 08:46:59 yes 08:47:03 that was last week 08:47:14 we may also notice that fuel, compass and joid are consistent (job naming) 08:47:15 I explained more or less all the steps they need to run functest in CI 08:47:20 it will be great if daisy could be 08:47:23 but so far I see only deployments 08:47:32 yes 08:47:37 https://build.opnfv.org/ci/job/daisy-deploy-baremetal-daily-master 08:47:40 the deployment is not stable yet 08:47:52 that link doesnt work for me 08:47:53 and of course if apex could also make some effort to be consistent it would be great... 08:47:57 neither for me 08:48:09 I think fdegir tried already :) 08:48:16 I know... 08:48:49 3 installers did it...it does not look impossible to do 08:48:51 ok, in the same direction, let´s see who can work with what 08:48:53 #topic Lab availability 08:48:54 and it is easier for us 08:49:20 I have a fuel environment where I can login that belongs to sdnvpn team (including myself) 08:49:31 it´s a virtual deployment, but it should be fine 08:49:42 #info jose_lausuch troubleshoot functest-fuel issues 08:49:54 who can work on apex? juhak ? 08:50:19 I can check apex issues 08:50:31 #info morgan_orange: compass/colorado (Orange Paris), fuel/colorado (Orange Paris), joid/Danube? (when available on Orange Lannion) and access to LF pod1 and LF POD2 shall still be OK (but not used for a long time and troubleshooting on CI pod is not easy...) 08:50:31 do you have hw to test on? 08:51:11 yes 08:51:24 ok perfect 08:51:29 what about the others? 08:51:34 #info juhak apex/ (Nokia pod) 08:51:34 but our apex env Colorado... 08:51:35 do you have testbeds? 08:51:42 #undo 08:51:42 Removing item from minutes: 08:51:45 we should be testing latest apex 08:51:48 latest everything 08:52:08 #info juhak apex/colorado (Nokia pod) 08:52:31 do you have a chance to test latest deployments ? 08:52:33 not always easy, In Paris some teams are using to perform tests on a "stable" version 08:52:49 at this moment can update to latest 08:52:52 so no plan to upgrade to Danube candidate... 08:52:57 ok 08:53:06 others are testing w/ Colodaro 08:53:28 ok 08:53:36 HelenYao: SerenaFeng do you have testbeds? 08:53:36 it is logical that community pods deal with stable version (i.e.colorado...) 08:53:42 who works on compass? 08:53:49 but we need community pod with danube candidate version for troubleshooting 08:53:53 I have compass mitaka and newton 08:53:59 ok 08:54:17 for joid, I guess we have to wait for intel pods? 08:54:21 #info HelenYao compass/colorado and compass/danube (Huawei pod) 08:54:35 jose_lausuch: yes and we will use one of the Orange POD 08:54:47 ok good 08:54:47 our apex deploy failed for now 08:55:11 so we are set, everyone has some hardware to test stuff on 08:55:42 and daisy? 08:55:44 can we troubleshoot it on virtual apex pod? 08:55:54 SerenaFeng: yes I think so 08:55:58 we have virtual apex pod 08:56:20 if you create an infra ticket, I could set up a pod with both installers, apex and fuel 08:56:28 using latest builds 08:56:35 and troubleshoot at least these 2 installers there 08:56:48 centos as op.system 08:57:00 virtual deployments 08:57:14 ok, assign the task to me, I will try to troubleshoot it 08:57:52 well, the task is for all :) 08:58:02 ok 08:58:04 I will give access to the few of us that work on it 08:58:07 ok 08:58:09 before we finish 08:58:17 #topic Gerrit gate 08:58:29 for daisy, I will follow their progress 08:58:32 maybe we need 1 virtual pod for functest for apex and fuel 08:58:53 Nikolas proposed me to use the same apex deployment as he does for the ODL-pipeline based on snapshots 08:58:54 I can create an infra ticket 08:59:02 it takes 5 minutes to deploy 08:59:06 ok, thanks 08:59:42 we could use that and have a verify job (20min max) that runs maybe prepare_env and 1 test case (healthechk?) 08:59:47 would you like to have something like that? 09:00:06 Am I wrong or is it already in place on apex? 09:00:23 it is 09:00:31 for apex verify jobs 09:00:33 I would suggest prepare env + healtcheck + a vPing 09:00:36 but not for functest verify jobs :) 09:00:40 ok 09:00:46 agree 09:00:57 it has to be something not depending on the installer of course 09:01:07 the prepare env is then maybe a bit heavy for what we want to do 09:01:09 because we will use apex snapshots 09:01:20 we could have something lighter...no need to install rally/tempest, clone all the repos... 09:01:22 it takes 10 seconds 09:01:28 but the challenge is 09:01:36 to build a docker container and put the patch changes in there 09:02:23 hmm if you run the git review -D in the docker 09:02:32 yes 09:02:39 so we need to do some hacking around :) 09:02:39 we can deploy the container by volume mapping 09:02:43 yes 09:02:47 git review is not support for now 09:02:47 thats also another idea 09:02:48 then reinstall the functest 09:03:03 so 2 setps 09:03:04 steps 09:03:13 docker volume including the local patch 09:03:19 and reinstall functest 09:03:30 sounds good 09:03:36 this would be pulling the latest container, without building it again 09:03:40 right? 09:04:01 the docker latest image should be pulled from time to time to embrace the latest change of other project 09:04:15 we can include the pull in the job 09:04:16