08:01:32 <joehuang> #startmeeting multisite
08:01:32 <collabot`> Meeting started Thu Feb 23 08:01:32 2017 UTC.  The chair is joehuang. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:01:32 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:01:32 <collabot`> The meeting name has been set to 'multisite'
08:01:54 <joehuang> hello, wait for dimitri for a while
08:01:59 <goutham> yea
08:03:06 <joehuang> #info joehuang
08:03:15 <goutham> #info goutha,
08:03:21 <sorantis> #info dimitrie
08:03:21 <goutham> #info goutham
08:03:28 <sorantis> #info dimitri
08:03:33 <sorantis> hi
08:03:36 <joehuang> hi
08:03:45 <goutham> hi all
08:04:23 <joehuang> the multisite CI is running normal now
08:04:48 <joehuang> https://build.opnfv.org/ci/view/multisite/builds
08:05:14 <sorantis> it actually isn't
08:05:19 <sorantis> https://build.opnfv.org/ci/view/multisite/job/functest-fuel-virtual-suite-master/259/console
08:05:20 <joehuang> but the test cases execution has not been done yet
08:05:27 <joehuang> yes
08:05:30 <sorantis> functest gives false positive
08:05:42 <sorantis> the jobs were done 2 weeks ago
08:05:54 <joehuang> the kingbird test were restored in this patch
08:06:03 <joehuang> and got merged this morning
08:06:22 <joehuang> #link https://gerrit.opnfv.org/gerrit/#/c/29003/
08:06:30 <sorantis> so docker hasn’t been updated yet
08:06:37 <joehuang> it worked before it was commented
08:07:06 <joehuang> don't know why the functest can't work after the uncomment
08:07:42 <sorantis> if it got merged just this morning (define morning?) I think it takes some time before the new docker image is built
08:07:58 <joehuang> yes, this morning
08:08:35 <joehuang> 6:13am my time, and the functest was executed around 8:am
08:09:24 <sorantis> I’ll have to login and check if the changes are part of the last docker run
08:09:54 <joehuang> when will the functest container image be built
08:10:03 <joehuang> ok, please
08:10:24 <joehuang> kingbird folder should be there if it's latest image
08:10:35 <joehuang> after the patch was merged
08:11:18 <sorantis> yes it was
08:11:20 <sorantis> still failed
08:12:00 <joehuang> you mean, the kingbird code was installed in the functest image?
08:12:45 <sorantis> ======
08:12:45 <sorantis> Totals
08:12:47 <sorantis> ======
08:12:48 <sorantis> Ran: 0 tests in 0.000 sec.
08:12:50 <sorantis> - Success: 0
08:12:51 <sorantis> - Skipped: 0
08:12:53 <sorantis> - Expected failures: 0
08:12:54 <sorantis> - Unexpected success: 0
08:12:54 <sorantis> - Failures: 0
08:13:09 <sorantis> nothing was executed
08:13:27 <joehuang> could you check whether the tempest cases could be disovered by tempest
08:13:33 <joehuang> just go to tempest folder
08:13:38 <joehuang> and run testr list-tests | grep kingbird
08:14:02 <goutham> use captial "K"ingbird
08:14:30 <joehuang> lower case should work too
08:14:50 <goutham> yea but "K" should list all the test-cases
08:15:01 <sorantis> running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
08:15:02 <sorantis> OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
08:15:03 <sorantis> OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
08:15:05 <sorantis> OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
08:15:06 <sorantis> ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
08:15:08 <sorantis> scenario.quota_management.client_tests.test_quota_class_api.KingbirdQuotaClassTestJSON.test_kb_quota_class_delete_method
08:15:09 <sorantis> scenario.quota_management.client_tests.test_quota_class_api.KingbirdQuotaClassTestJSON.test_kb_quota_class_get_method
08:15:11 <sorantis> scenario.quota_management.client_tests.test_quota_class_api.KingbirdQuotaClassTestJSON.test_kb_quota_class_get_method_wrong_class_name
08:15:12 <sorantis> scenario.quota_management.client_tests.test_quota_class_api.KingbirdQuotaClassTestJSON.test_kb_quota_class_put_method
08:15:14 <sorantis> scenario.quota_management.client_tests.test_quota_class_api.KingbirdQuotaClassTestJSON.test_kb_quota_class_wrong_input
08:15:15 <sorantis> scenario.quota_management.client_tests.test_quota_class_api.KingbirdQuotaClassTestJSON.test_kb_quota_class_wrong_quotas
08:15:17 <sorantis> scenario.quota_management.client_tests.test_quota_class_api.KingbirdQuotaClassTestJSON.test_kb_quota_default_class_get_method
08:15:18 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_get_quota_usage_for_project
08:15:20 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_kingbird_delete_all_method
08:15:21 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_kingbird_delete_method
08:15:23 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_kingbird_get_default_method_after_delete
08:15:24 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_kingbird_get_default_method_after_update
08:15:26 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_kingbird_get_method
08:15:27 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_kingbird_put_method
08:15:29 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_kingbird_put_method_wrong_token
08:15:30 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_quota_exceed_after_sync
08:15:31 <sorantis> scenario.quota_management.client_tests.test_quota_management_api.KingbirdQMTestJSON.test_quota_sync_for_project
08:15:32 <sorantis> I found this in the test_list.txt
08:15:36 <sorantis> root@4a390c4679cf:~/functest/results/tempest# ll
08:15:36 <sorantis> total 20K
08:15:38 <sorantis> -rw-r--r-- 1 root root  141 Feb 23 03:07 environment.log
08:15:39 <sorantis> -rw-r--r-- 1 root root    0 Feb 23 03:07 tempest-error.log
08:15:41 <sorantis> -rw-r--r-- 1 root root 2.4K Feb 23 03:07 tempest.conf
08:15:42 <sorantis> -rw-r--r-- 1 root root  729 Feb 23 03:07 tempest.log
08:15:43 <sorantis> -rw-r--r-- 1 root root 2.3K Feb 23 03:07 test_list.txt
08:15:44 <sorantis> -rw-r--r-- 1 root root 2.3K Feb 23 03:07 test_raw_list.txt
08:16:36 <joehuang> so test cases are there, no problem
08:16:52 <joehuang> what's the information in tempest-error.log
08:17:55 <sorantis> -rw-r--r-- 1 root root    0 Feb 23 03:07 tempest-error.log
08:17:58 <sorantis> it’s empty
08:18:49 <joehuang> strange, that means the test not triggered
08:19:18 <joehuang> how about tempest.log
08:20:12 <sorantis> I already showed the output
08:20:14 <sorantis> 2017-02-23 03:07:31.812 663 INFO rally.api [-] Starting verification (UUID=d8c216f2-cf3b-4dbc-8934-f5cec87a7e6f) for deployment 'opnfv-rally' (UUID=810e9797-debd-4fda-8909-756d7d646782) by verifier 'opnfv-tempest' (UUID=f62eb419-bc0b-4fd0-982b-0094396452eb).
08:20:15 <sorantis> 2017-02-23 03:07:37.531 663 INFO rally.api [-] Verification (UUID=d8c216f2-cf3b-4dbc-8934-f5cec87a7e6f) has been successfully finished for deployment 'opnfv-rally' (UUID=810e9797-debd-4fda-8909-756d7d646782)!
08:20:16 <sorantis> ======
08:20:18 <sorantis> Totals
08:20:19 <sorantis> ======
08:20:21 <sorantis> Ran: 0 tests in 0.000 sec.
08:20:22 <sorantis> - Success: 0
08:20:24 <sorantis> - Skipped: 0
08:20:25 <sorantis> - Expected failures: 0
08:20:26 <sorantis> - Unexpected success: 0
08:20:27 <sorantis> - Failures: 0
08:20:28 <sorantis> Using verification (UUID=d8c216f2-cf3b-4dbc-8934-f5cec87a7e6f) as the default verification for the future operations
08:20:36 <joehuang> ok, nothing executed
08:20:39 <sorantis> in tempest.conf
08:20:42 <sorantis> [kingbird]
08:20:42 <sorantis> endpoint_type = publicURL
08:20:43 <sorantis> time_to_sync = 20
08:20:44 <sorantis> endpoint_url = http://192.168.0.3:8118/
08:20:45 <sorantis> api_version = http://172.16.0.3:8118/v1.0
08:20:46 <sorantis> what generates this part?
08:21:10 <joehuang> is the endpoint correct
08:21:27 <sorantis> root@node-4:~# openstack endpoint list | grep kingbird
08:21:27 <sorantis> | 957f52529d314a839a50bec3aa7869bf | RegionOne | kingbird       | multisite      | True    | internal  | http://192.168.0.2:8118/v1.0                  |
08:21:28 <sorantis> | ace7162eb2a84287af63db9e041718c3 | RegionOne | kingbird       | multisite      | True    | admin     | http://192.168.0.2:8118/v1.0                  |
08:21:29 <sorantis> | cf9947e65a9b434694588b6203f0f294 | RegionOne | kingbird       | multisite      | True    | public    | http://172.16.0.3:8118/v1.0                   |
08:21:29 <joehuang> the api version is full path
08:22:12 <joehuang> endpoint url is the internal/admin url
08:22:22 <joehuang> but the api_version is public url
08:22:33 <sorantis> the ip addresses don’t match
08:22:44 <sorantis> again, what generates this?
08:22:58 <joehuang> funtest generated it
08:23:22 <joehuang> ip address shifted
08:23:25 <sorantis> based on what?
08:23:35 <joehuang> do you have multiple server
08:23:43 <sorantis> ?
08:24:09 <joehuang> what's the IP address of regionone and regiontwo host
08:25:55 <joehuang> ./opnfv_tests/openstack/tempest/conf_utils.py:        kingbird_endpoint_url = "http://%s:%s/" % (bind_host, bind_port) ./opnfv_tests/openstack/tempest/conf_utils.py:        # kingbird_endpoint_url = os.popen(cmd).read() ./opnfv_tests/openstack/tempest/conf_utils.py:        kingbird_endpoint_url = os_utils.get_endpoint(service_type='kingbird') ./opnfv_tests/openstack/tempest/conf_utils.py:    config.set('kingbird', 'endpoint
08:26:49 <joehuang> in this function def configure_tempest_multisite_params(tempest_conf_file): of the file opnfv_tests/openstack/tempest/conf_utils.py
08:28:32 <sorantis> the ip address is fetched from regionTwo, not regionOne
08:29:07 <joehuang> The script need to update, get the endpoint url from endpoint lisr directly but not from the bind_host, bind_port, this was previous environment setup
08:29:40 <joehuang> do you remember in the old functest (preconfigured one)
08:30:11 <joehuang> the script ssh into the host, and access the /etc/kingbird/kingbird.conf
08:30:25 <sorantis> weird. functest is triggered in regionOne, why would it fetch info from regionTwo?
08:30:35 <joehuang> these codes need to be updated
08:30:43 <joehuang> one second, I'll show you the code
08:32:03 <joehuang> #link https://github.com/opnfv/functest/blob/6c8b09ca1184139a784a7d6c42e0ca09fa57dca9/functest/opnfv_tests/openstack/tempest/conf_utils.py#L208-L278
08:32:19 <joehuang> this function has to be updated too for the new environment
08:33:09 <sorantis> kingbird_api_version = os_utils.get_endpoint(service_type='multisite')
08:33:17 <sorantis> service_type is wrong
08:33:21 <joehuang> but endpoint url
08:33:22 <sorantis> it should be kingbird
08:33:57 <joehuang> is obtained through ssh
08:34:37 <sorantis> ok, that’s really weird
08:34:38 <sorantis> | 957f52529d314a839a50bec3aa7869bf | RegionOne | kingbird       | multisite      | True    | internal  | http://192.168.0.2:8118/v1.0                  |
08:34:38 <sorantis> | ace7162eb2a84287af63db9e041718c3 | RegionOne | kingbird       | multisite      | True    | admin     | http://192.168.0.2:8118/v1.0                  |
08:34:39 <sorantis> | cf9947e65a9b434694588b6203f0f294 | RegionOne | kingbird       | multisite      | True    | public    | http://172.16.0.3:8118/v1.0                   |
08:34:49 <sorantis> I remember chaning the service type for kingbird
08:35:23 <sorantis> nope
08:35:24 <sorantis> openstack service create --name=kingbird --description="Kingbird" multisite
08:35:26 <sorantis> i didn’t
08:36:14 <joehuang> if CI_INSTALLER_TYPE == 'fuel': ..........        kingbird_endpoint_url = "http://%s:%s/" % (bind_host, bind_port)
08:36:42 <joehuang> the code is dedicated for the old preconfigured envirment
08:39:33 <sorantis> http://192.168.0.2:5000
08:39:47 <sorantis> OS_AUTH_URL='http://192.168.0.2:5000/'
08:40:24 <sorantis> [DEFAULT]
08:40:24 <sorantis> bind_port = 8118
08:40:25 <sorantis> bind_host = 192.168.0.3
08:40:50 <sorantis> so if this code fetches bind_host and port, then it’s fetching wrong values I think
08:41:21 <sorantis> bind_host=$(sed -n 's/^admin_bind_host *= *\([^ ]*.*\)/\1/p' < /etc/keystone/keystone.conf)
08:41:27 <sorantis> that’s what we’re writing there
08:41:46 <sorantis> I don’t know what was the reason for such a fetish
08:41:52 <sorantis> but there was one
08:43:17 <sorantis> let’s roll back
08:43:32 <joehuang> kingbird_endpoint_url = os_utils.get_endpoint(service_type='kingbird') should be enough
08:43:40 <joehuang> ok, would you submit a patch
08:43:46 <sorantis> if there was a connectivity problem, then the logs would show something, right? like connection timeout
08:43:54 <sorantis> but the error log is clear
08:44:07 <sorantis> looks like not not a single test gets executed
08:45:05 <sorantis> I have 5 more minutes, before I run to another meeting.
08:45:12 <joehuang> you mean even the test has not been triggered?
08:45:56 <sorantis> ======
08:45:56 <sorantis> Totals
08:45:58 <sorantis> ======
08:45:59 <sorantis> Ran: 0 tests in 0.000 sec.
08:46:01 <sorantis> - Success: 0
08:46:02 <sorantis> - Skipped: 0
08:46:04 <sorantis> - Expected failures: 0
08:46:05 <sorantis> - Unexpected success: 0
08:46:05 <sorantis> - Failures: 0
08:46:09 <sorantis> There are no failed test cases
08:46:46 <sorantis> oh, the meeting got shifted. I have more time :)
08:46:47 <joehuang> what's the suggestion from Goutham
08:46:54 <joehuang> great
08:47:03 <sorantis> I don’t think Goutham knows this part
08:47:11 <sorantis> it was mei mei who worked on functest
08:47:29 <goutham> yes i am seeing these for the first for the next release i will take care of this as well
08:47:37 <joehuang> ok, I'll contact mei mei. Is this code written by meimei
08:48:01 <goutham> ok
08:48:39 <sorantis> I don’t know who updated this code last
08:49:01 <sorantis> I don’t recognize it tbh
08:49:11 <joehuang> Juha Kosonen, 8 days ago
08:49:27 <sorantis> Juha didn’t write this part
08:49:46 <sorantis> I think he only made sure that multisite was selected by functest
08:49:49 <joehuang> https://github.com/opnfv/functest/commit/bc4c0c3cc4beea6b86a5f06663323c654385a04e
08:49:51 <sorantis> even that didn’t work before
08:49:59 <sorantis> fucntest would trigger ALL test cases
08:50:26 <sorantis> ok
08:51:19 <joehuang> but currently seems not trigger at all
08:51:30 <sorantis> yes
08:53:00 <joehuang> I think there are two issues here
08:53:23 <joehuang> 1) kingbird functest test cases triggering
08:54:09 <joehuang> 2) endpoint_url/api_version address is incorrect
08:54:31 <sorantis> I’m not sure about the second point. We don’t know if it’s correct or not
08:54:45 <sorantis> the first issue blocks us from seeing further test output
08:55:44 <joehuang> from your endpoint list, the kingbird is running on 192.168.0.3
08:56:21 <joehuang> sorry, 192.168.0.2 instead
08:56:26 <sorantis> yes, but this url was taken from keyston bind host
08:56:37 <joehuang> public  public    | http://172.16.0.3:8118/v1.0
08:57:24 <joehuang> openstack endpoint list will sure where the kingbird is registered
08:57:51 <joehuang> could you check to see if the kingbird processes running in that host or not
09:00:22 <sorantis> it’s not!
09:00:32 <sorantis> but I was running
09:00:50 <sorantis> ++ sshpass -p r00tme ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@10.20.0.2 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no 10.20.0.6 "pgrep -f kingbird-engine || echo dead"'
09:00:50 <sorantis> + engine_pid=29388
09:00:52 <sorantis> ++ sshpass -p r00tme ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@10.20.0.2 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no 10.20.0.6 "pgrep -f kingbird-api || echo dead"'
09:00:53 <sorantis> + api_pid=29411
09:00:55 <sorantis> + '[' 29388 == dead ']'
09:00:56 <sorantis> + '[' 29411 == dead ']'
09:00:56 <sorantis> + echo 'Deployment complete!
09:01:00 <sorantis> something is going on with the nodes
09:01:07 <joehuang> the endpoint in keystone is registered through the scripts https://git.opnfv.org/multisite/tree/tools/kingbird/install_kingbird.sh
09:01:09 <joehuang> ok
09:01:33 <joehuang> please check after the meeting.
09:02:15 <sorantis> I’ll check that part, but can you check with functest
09:02:41 <joehuang> I think the tempest configuration of kingbird should be obtained from openstack endpoint list
09:02:58 <joehuang> I'll contact Joha and Meimei about the functest
09:03:20 <sorantis> before that we need to know why no tests get executed
09:03:23 <joehuang> ok, thank you for attending the meeting
09:03:30 <joehuang> time is up
09:03:32 <sorantis> ok
09:03:33 <sorantis> thanks
09:03:40 <goutham> ok bye all
09:03:43 <sorantis> bye
09:03:45 <joehuang> #endmeeting