08:01:32 <joehuang> #startmeeting multisite 08:01:32 <collabot`> Meeting started Thu Feb 23 08:01:32 2017 UTC. The chair is joehuang. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:01:32 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic. 08:01:32 <collabot`> The meeting name has been set to 'multisite' 08:01:54 <joehuang> hello, wait for dimitri for a while 08:01:59 <goutham> yea 08:03:06 <joehuang> #info joehuang 08:03:15 <goutham> #info goutha, 08:03:21 <sorantis> #info dimitrie 08:03:21 <goutham> #info goutham 08:03:28 <sorantis> #info dimitri 08:03:33 <sorantis> hi 08:03:36 <joehuang> hi 08:03:45 <goutham> hi all 08:04:23 <joehuang> the multisite CI is running normal now 08:04:48 <joehuang> https://build.opnfv.org/ci/view/multisite/builds 08:05:14 <sorantis> it actually isn't 08:05:19 <sorantis> https://build.opnfv.org/ci/view/multisite/job/functest-fuel-virtual-suite-master/259/console 08:05:20 <joehuang> but the test cases execution has not been done yet 08:05:27 <joehuang> yes 08:05:30 <sorantis> functest gives false positive 08:05:42 <sorantis> the jobs were done 2 weeks ago 08:05:54 <joehuang> the kingbird test were restored in this patch 08:06:03 <joehuang> and got merged this morning 08:06:22 <joehuang> #link https://gerrit.opnfv.org/gerrit/#/c/29003/ 08:06:30 <sorantis> so docker hasn’t been updated yet 08:06:37 <joehuang> it worked before it was commented 08:07:06 <joehuang> don't know why the functest can't work after the uncomment 08:07:42 <sorantis> if it got merged just this morning (define morning?) I think it takes some time before the new docker image is built 08:07:58 <joehuang> yes, this morning 08:08:35 <joehuang> 6:13am my time, and the functest was executed around 8:am 08:09:24 <sorantis> I’ll have to login and check if the changes are part of the last docker run 08:09:54 <joehuang> when will the functest container image be built 08:10:03 <joehuang> ok, please 08:10:24 <joehuang> kingbird folder should be there if it's latest image 08:10:35 <joehuang> after the patch was merged 08:11:18 <sorantis> yes it was 08:11:20 <sorantis> still failed 08:12:00 <joehuang> you mean, the kingbird code was installed in the functest image? 08:12:45 <sorantis> ====== 08:12:45 <sorantis> Totals 08:12:47 <sorantis> ====== 08:12:48 <sorantis> Ran: 0 tests in 0.000 sec. 08:12:50 <sorantis> - Success: 0 08:12:51 <sorantis> - Skipped: 0 08:12:53 <sorantis> - Expected failures: 0 08:12:54 <sorantis> - Unexpected success: 0 08:12:54 <sorantis> - Failures: 0 08:13:09 <sorantis> nothing was executed 08:13:27 <joehuang> could you check whether the tempest cases could be disovered by tempest 08:13:33 <joehuang> just go to tempest folder 08:13:38 <joehuang> and run testr list-tests | grep kingbird 08:14:02 <goutham> use captial "K"ingbird 08:14:30 <joehuang> lower case should work too 08:14:50 <goutham> yea but "K" should list all the test-cases 08:15:01 <sorantis> running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ 08:15:02 <sorantis> OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ 08:15:03 <sorantis> OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \ 08:15:05 <sorantis> OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \ 08:15:06 <sorantis> ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list 08:15:08 <sorantis> scenario.quota_management.client_tests.test_quota_class_api.KingbirdQuotaClassTestJSON.test_kb_quota_class_delete_method 08:15:09 <sorantis> scenario.quota_management.client_tests.test_quota_class_api.KingbirdQuotaClassTestJSON.test_kb_quota_class_get_method 08:15:11 <sorantis> scenario.quota_management.client_tests.test_quota_class_api.KingbirdQuotaClassTestJSON.test_kb_quota_class_get_method_wrong_class_name 08:15:12 <sorantis> scenario.quota_management.client_tests.test_quota_class_api.KingbirdQuotaClassTestJSON.test_kb_quota_class_put_method 08:15:14 <sorantis> scenario.quota_management.client_tests.test_quota_class_api.KingbirdQuotaClassTestJSON.test_kb_quota_class_wrong_input 08:15:15 <sorantis> scenario.quota_management.client_tests.test_quota_class_api.KingbirdQuotaClassTestJSON.test_kb_quota_class_wrong_quotas 08:15:17 <sorantis> scenario.quota_management.client_tests.test_quota_class_api.KingbirdQuotaClassTestJSON.test_kb_quota_default_class_get_method 08:15:18 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_get_quota_usage_for_project 08:15:20 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_kingbird_delete_all_method 08:15:21 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_kingbird_delete_method 08:15:23 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_kingbird_get_default_method_after_delete 08:15:24 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_kingbird_get_default_method_after_update 08:15:26 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_kingbird_get_method 08:15:27 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_kingbird_put_method 08:15:29 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_kingbird_put_method_wrong_token 08:15:30 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_quota_exceed_after_sync 08:15:31 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_quota_sync_for_project 08:15:32 <sorantis> I found this in the test_list.txt 08:15:36 <sorantis> root@4a390c4679cf:~/functest/results/tempest# ll 08:15:36 <sorantis> total 20K 08:15:38 <sorantis> -rw-r--r-- 1 root root 141 Feb 23 03:07 environment.log 08:15:39 <sorantis> -rw-r--r-- 1 root root 0 Feb 23 03:07 tempest-error.log 08:15:41 <sorantis> -rw-r--r-- 1 root root 2.4K Feb 23 03:07 tempest.conf 08:15:42 <sorantis> -rw-r--r-- 1 root root 729 Feb 23 03:07 tempest.log 08:15:43 <sorantis> -rw-r--r-- 1 root root 2.3K Feb 23 03:07 test_list.txt 08:15:44 <sorantis> -rw-r--r-- 1 root root 2.3K Feb 23 03:07 test_raw_list.txt 08:16:36 <joehuang> so test cases are there, no problem 08:16:52 <joehuang> what's the information in tempest-error.log 08:17:55 <sorantis> -rw-r--r-- 1 root root 0 Feb 23 03:07 tempest-error.log 08:17:58 <sorantis> it’s empty 08:18:49 <joehuang> strange, that means the test not triggered 08:19:18 <joehuang> how about tempest.log 08:20:12 <sorantis> I already showed the output 08:20:14 <sorantis> 2017-02-23 03:07:31.812 663 INFO rally.api [-] Starting verification (UUID=d8c216f2-cf3b-4dbc-8934-f5cec87a7e6f) for deployment 'opnfv-rally' (UUID=810e9797-debd-4fda-8909-756d7d646782) by verifier 'opnfv-tempest' (UUID=f62eb419-bc0b-4fd0-982b-0094396452eb). 08:20:15 <sorantis> 2017-02-23 03:07:37.531 663 INFO rally.api [-] Verification (UUID=d8c216f2-cf3b-4dbc-8934-f5cec87a7e6f) has been successfully finished for deployment 'opnfv-rally' (UUID=810e9797-debd-4fda-8909-756d7d646782)! 08:20:16 <sorantis> ====== 08:20:18 <sorantis> Totals 08:20:19 <sorantis> ====== 08:20:21 <sorantis> Ran: 0 tests in 0.000 sec. 08:20:22 <sorantis> - Success: 0 08:20:24 <sorantis> - Skipped: 0 08:20:25 <sorantis> - Expected failures: 0 08:20:26 <sorantis> - Unexpected success: 0 08:20:27 <sorantis> - Failures: 0 08:20:28 <sorantis> Using verification (UUID=d8c216f2-cf3b-4dbc-8934-f5cec87a7e6f) as the default verification for the future operations 08:20:36 <joehuang> ok, nothing executed 08:20:39 <sorantis> in tempest.conf 08:20:42 <sorantis> [kingbird] 08:20:42 <sorantis> endpoint_type = publicURL 08:20:43 <sorantis> time_to_sync = 20 08:20:44 <sorantis> endpoint_url = http://192.168.0.3:8118/ 08:20:45 <sorantis> api_version = http://172.16.0.3:8118/v1.0 08:20:46 <sorantis> what generates this part? 08:21:10 <joehuang> is the endpoint correct 08:21:27 <sorantis> root@node-4:~# openstack endpoint list | grep kingbird 08:21:27 <sorantis> | 957f52529d314a839a50bec3aa7869bf | RegionOne | kingbird | multisite | True | internal | http://192.168.0.2:8118/v1.0 | 08:21:28 <sorantis> | ace7162eb2a84287af63db9e041718c3 | RegionOne | kingbird | multisite | True | admin | http://192.168.0.2:8118/v1.0 | 08:21:29 <sorantis> | cf9947e65a9b434694588b6203f0f294 | RegionOne | kingbird | multisite | True | public | http://172.16.0.3:8118/v1.0 | 08:21:29 <joehuang> the api version is full path 08:22:12 <joehuang> endpoint url is the internal/admin url 08:22:22 <joehuang> but the api_version is public url 08:22:33 <sorantis> the ip addresses don’t match 08:22:44 <sorantis> again, what generates this? 08:22:58 <joehuang> funtest generated it 08:23:22 <joehuang> ip address shifted 08:23:25 <sorantis> based on what? 08:23:35 <joehuang> do you have multiple server 08:23:43 <sorantis> ? 08:24:09 <joehuang> what's the IP address of regionone and regiontwo host 08:25:55 <joehuang> ./opnfv_tests/openstack/tempest/conf_utils.py: kingbird_endpoint_url = "http://%s:%s/" % (bind_host, bind_port) ./opnfv_tests/openstack/tempest/conf_utils.py: # kingbird_endpoint_url = os.popen(cmd).read() ./opnfv_tests/openstack/tempest/conf_utils.py: kingbird_endpoint_url = os_utils.get_endpoint(service_type='kingbird') ./opnfv_tests/openstack/tempest/conf_utils.py: config.set('kingbird', 'endpoint 08:26:49 <joehuang> in this function def configure_tempest_multisite_params(tempest_conf_file): of the file opnfv_tests/openstack/tempest/conf_utils.py 08:28:32 <sorantis> the ip address is fetched from regionTwo, not regionOne 08:29:07 <joehuang> The script need to update, get the endpoint url from endpoint lisr directly but not from the bind_host, bind_port, this was previous environment setup 08:29:40 <joehuang> do you remember in the old functest (preconfigured one) 08:30:11 <joehuang> the script ssh into the host, and access the /etc/kingbird/kingbird.conf 08:30:25 <sorantis> weird. functest is triggered in regionOne, why would it fetch info from regionTwo? 08:30:35 <joehuang> these codes need to be updated 08:30:43 <joehuang> one second, I'll show you the code 08:32:03 <joehuang> #link https://github.com/opnfv/functest/blob/6c8b09ca1184139a784a7d6c42e0ca09fa57dca9/functest/opnfv_tests/openstack/tempest/conf_utils.py#L208-L278 08:32:19 <joehuang> this function has to be updated too for the new environment 08:33:09 <sorantis> kingbird_api_version = os_utils.get_endpoint(service_type='multisite') 08:33:17 <sorantis> service_type is wrong 08:33:21 <joehuang> but endpoint url 08:33:22 <sorantis> it should be kingbird 08:33:57 <joehuang> is obtained through ssh 08:34:37 <sorantis> ok, that’s really weird 08:34:38 <sorantis> | 957f52529d314a839a50bec3aa7869bf | RegionOne | kingbird | multisite | True | internal | http://192.168.0.2:8118/v1.0 | 08:34:38 <sorantis> | ace7162eb2a84287af63db9e041718c3 | RegionOne | kingbird | multisite | True | admin | http://192.168.0.2:8118/v1.0 | 08:34:39 <sorantis> | cf9947e65a9b434694588b6203f0f294 | RegionOne | kingbird | multisite | True | public | http://172.16.0.3:8118/v1.0 | 08:34:49 <sorantis> I remember chaning the service type for kingbird 08:35:23 <sorantis> nope 08:35:24 <sorantis> openstack service create --name=kingbird --description="Kingbird" multisite 08:35:26 <sorantis> i didn’t 08:36:14 <joehuang> if CI_INSTALLER_TYPE == 'fuel': .......... kingbird_endpoint_url = "http://%s:%s/" % (bind_host, bind_port) 08:36:42 <joehuang> the code is dedicated for the old preconfigured envirment 08:39:33 <sorantis> http://192.168.0.2:5000 08:39:47 <sorantis> OS_AUTH_URL='http://192.168.0.2:5000/' 08:40:24 <sorantis> [DEFAULT] 08:40:24 <sorantis> bind_port = 8118 08:40:25 <sorantis> bind_host = 192.168.0.3 08:40:50 <sorantis> so if this code fetches bind_host and port, then it’s fetching wrong values I think 08:41:21 <sorantis> bind_host=$(sed -n 's/^admin_bind_host *= *\([^ ]*.*\)/\1/p' < /etc/keystone/keystone.conf) 08:41:27 <sorantis> that’s what we’re writing there 08:41:46 <sorantis> I don’t know what was the reason for such a fetish 08:41:52 <sorantis> but there was one 08:43:17 <sorantis> let’s roll back 08:43:32 <joehuang> kingbird_endpoint_url = os_utils.get_endpoint(service_type='kingbird') should be enough 08:43:40 <joehuang> ok, would you submit a patch 08:43:46 <sorantis> if there was a connectivity problem, then the logs would show something, right? like connection timeout 08:43:54 <sorantis> but the error log is clear 08:44:07 <sorantis> looks like not not a single test gets executed 08:45:05 <sorantis> I have 5 more minutes, before I run to another meeting. 08:45:12 <joehuang> you mean even the test has not been triggered? 08:45:56 <sorantis> ====== 08:45:56 <sorantis> Totals 08:45:58 <sorantis> ====== 08:45:59 <sorantis> Ran: 0 tests in 0.000 sec. 08:46:01 <sorantis> - Success: 0 08:46:02 <sorantis> - Skipped: 0 08:46:04 <sorantis> - Expected failures: 0 08:46:05 <sorantis> - Unexpected success: 0 08:46:05 <sorantis> - Failures: 0 08:46:09 <sorantis> There are no failed test cases 08:46:46 <sorantis> oh, the meeting got shifted. I have more time :) 08:46:47 <joehuang> what's the suggestion from Goutham 08:46:54 <joehuang> great 08:47:03 <sorantis> I don’t think Goutham knows this part 08:47:11 <sorantis> it was mei mei who worked on functest 08:47:29 <goutham> yes i am seeing these for the first for the next release i will take care of this as well 08:47:37 <joehuang> ok, I'll contact mei mei. Is this code written by meimei 08:48:01 <goutham> ok 08:48:39 <sorantis> I don’t know who updated this code last 08:49:01 <sorantis> I don’t recognize it tbh 08:49:11 <joehuang> Juha Kosonen, 8 days ago 08:49:27 <sorantis> Juha didn’t write this part 08:49:46 <sorantis> I think he only made sure that multisite was selected by functest 08:49:49 <joehuang> https://github.com/opnfv/functest/commit/bc4c0c3cc4beea6b86a5f06663323c654385a04e 08:49:51 <sorantis> even that didn’t work before 08:49:59 <sorantis> fucntest would trigger ALL test cases 08:50:26 <sorantis> ok 08:51:19 <joehuang> but currently seems not trigger at all 08:51:30 <sorantis> yes 08:53:00 <joehuang> I think there are two issues here 08:53:23 <joehuang> 1) kingbird functest test cases triggering 08:54:09 <joehuang> 2) endpoint_url/api_version address is incorrect 08:54:31 <sorantis> I’m not sure about the second point. We don’t know if it’s correct or not 08:54:45 <sorantis> the first issue blocks us from seeing further test output 08:55:44 <joehuang> from your endpoint list, the kingbird is running on 192.168.0.3 08:56:21 <joehuang> sorry, 192.168.0.2 instead 08:56:26 <sorantis> yes, but this url was taken from keyston bind host 08:56:37 <joehuang> public public | http://172.16.0.3:8118/v1.0 08:57:24 <joehuang> openstack endpoint list will sure where the kingbird is registered 08:57:51 <joehuang> could you check to see if the kingbird processes running in that host or not 09:00:22 <sorantis> it’s not! 09:00:32 <sorantis> but I was running 09:00:50 <sorantis> ++ sshpass -p r00tme ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@10.20.0.2 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no 10.20.0.6 "pgrep -f kingbird-engine || echo dead"' 09:00:50 <sorantis> + engine_pid=29388 09:00:52 <sorantis> ++ sshpass -p r00tme ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@10.20.0.2 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no 10.20.0.6 "pgrep -f kingbird-api || echo dead"' 09:00:53 <sorantis> + api_pid=29411 09:00:55 <sorantis> + '[' 29388 == dead ']' 09:00:56 <sorantis> + '[' 29411 == dead ']' 09:00:56 <sorantis> + echo 'Deployment complete! 09:01:00 <sorantis> something is going on with the nodes 09:01:07 <joehuang> the endpoint in keystone is registered through the scripts https://git.opnfv.org/multisite/tree/tools/kingbird/install_kingbird.sh 09:01:09 <joehuang> ok 09:01:33 <joehuang> please check after the meeting. 09:02:15 <sorantis> I’ll check that part, but can you check with functest 09:02:41 <joehuang> I think the tempest configuration of kingbird should be obtained from openstack endpoint list 09:02:58 <joehuang> I'll contact Joha and Meimei about the functest 09:03:20 <sorantis> before that we need to know why no tests get executed 09:03:23 <joehuang> ok, thank you for attending the meeting 09:03:30 <joehuang> time is up 09:03:32 <sorantis> ok 09:03:33 <sorantis> thanks 09:03:40 <goutham> ok bye all 09:03:43 <sorantis> bye 09:03:45 <joehuang> #endmeeting