08:12:34 #startmeeting multisite 08:12:34 Meeting started Thu Aug 20 08:12:34 2015 UTC. The chair is joehuang. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:12:34 Useful Commands: #action #agreed #help #info #idea #link #topic. 08:12:34 The meeting name has been set to 'multisite' 08:12:46 #topic rollcall 08:12:52 #info joehuang 08:13:22 #info Malla 08:13:25 #info hafe 08:14:06 #topic usecase 1 verfication 08:14:13 #info Dimitri 08:14:27 hello, a short summary of the verification 08:14:38 the test is started by hafe 08:15:10 and the first step is to verify the cluster async. replication 08:16:03 the DB aync. replication between cluster works 08:16:28 I also got master replication between galera cluster to work 08:16:38 but hafe found that the token validation request was sent to the first keystone always 08:16:40 clusters 08:16:56 yeah seems to be a keystonemiddleware issue 08:17:12 i debuged into the source code 08:17:27 found that there is some issues in the source code 08:18:26 But even when I make the keystonemiddleware send the token validation to the right keystone server 08:19:02 the second keystone server did not validate the token correctly 08:19:54 I did not have a lot of time to find out why, but I found that the token validation is sent to the http://keyston IP: 5000/ 08:20:14 which is wrong, should be sent to the admin port 08:20:18 not the public one 08:20:19 #info Jun Li 08:20:22 yes 08:20:26 correct 08:20:36 but I tested and it does not help 08:20:57 in my own environment which is used to correct the bug in keystonemiddleware 08:21:11 should send an email to openstack(-dev) and ask 08:21:18 the token validation request is sent to http://keyston IP: 35357/ 08:22:02 that works in my own enviroment. will send a mail after more investigation 08:22:29 ok great 08:22:54 I am sure this can be fixed, can we look at it from an arch point of view? 08:23:25 And from the verification, I proposal to use a star mode " a cluster " with lot of "async slaves" 08:23:39 to hafe: agree 08:23:57 just to be clear; 08:24:19 how keystone is deployed and how the database is replicated is just characteristics 08:24:31 no (?) functional impact 08:24:33 KeyStone server with the cluster of DB will be deployed in two sites 08:25:15 correct? 08:25:24 for site level redundancy to provide write capability 08:26:11 and all other slaves which is async-replicated with DB will work as read-only keystone server 08:26:41 and in other sites, two or more slaves could be deployed for redundancy purpose 08:28:59 for functional impact, slave sites will be only used for token validation 08:29:59 but for create/update/delete function on KeyStone, should be done on the master sites: the two sites with DB cluster. 08:30:23 Is it more clear now? 08:32:18 Hi, Hafe, you mentioned write/read split in DB? could you describe in more detail? 08:33:09 well I am no database expert but clients can in scale out solution have different endpoints for write and read 08:33:30 client in this case is keystone 08:33:42 it has a single DB endpoint 08:34:18 so keystone in "slave" region cannot mutate any data 08:34:52 that restriction could be lifted with a write/read split 08:35:38 will the keystone in "slave" region access the db in local site or remote site? 08:35:52 local site 08:36:08 during token validation there will be db reads that you want to be local 08:36:43 yes 08:37:11 but again all this is just characteristics, no function impact 08:37:29 you can have a central keystone server that all sites use 08:37:37 and deal with faults with that 08:38:35 the db configuration in slave region keystones can be configured separately 08:39:11 what do you mean? 08:40:32 "you can have a central keystone server that all sites use" what does the central keystone server stand for 08:41:46 one keystone installation at a central site 08:42:21 how to disable the write in the "slave" region, through API policy or at DB level 08:43:24 right now DB level 08:43:53 the keystone DB user should probably not have write access 08:43:59 on slaves 08:44:14 hafe, not follow your idea for the central keystone. What's the main purpose for the central installation 08:44:19 just to hinder accidental writes by some admin or so 08:45:09 what I try to say is that how keystone and its db is deployed only affects characteristics 08:45:21 not functional impact really 08:45:47 you could have a single keystone server instance running in China serving all regions 08:45:58 still works until some links goes down 08:45:58 ok. understand, no new feature required 08:46:18 we are just looking at improving characteristics 08:46:31 in terms of keystone API latency and HA 08:46:40 site level failures 08:47:42 not even see one line code in the multisite git repo, why so hot discussion here 08:47:59 agree, for keystone, multisite project focus on how to deploy keystone 08:48:20 we don't need to put too much effort on it 08:48:23 no 08:48:43 the prototype was just an idea on how it can be done by the installer which is outside our scope 08:48:52 yes 08:49:11 but what about ETSI NFV and this matter? 08:49:21 are there any related reqs? 08:51:24 the prototype is to verify that keystone can work in multisite scenario with site level reliability, If don't, new feature required in keystone. 08:52:12 when we prioritized the use case, most of people put the identity in the 1st priority. 08:52:54 so site level reliability implies something like in the prototype? 08:53:02 yes 08:53:18 if each site with its own keystone server 08:53:37 then even this site failure, other site can still work with identity service 08:54:19 if fernet works. we have nothing to do in indentity management for multisite scenario 08:54:19 well it has more todo with management/orchestration I guess? 08:55:35 management/orchestration still rely on keystone for openstack identity managment and token validation 08:55:50 especially token validation 08:57:26 if the site where keystone server installed for the token validation is failed, and other sites will not able to handle all requests which use the keystone server for token validation 08:58:51 multi site == multi region cloud ? 08:59:16 single cloud, multiple region 08:59:17 currently all keystone deployment possibility can not guarranty site level keystone avaialbiltiy except fernet token based keystone installed in each site 08:59:46 single cloud, multi-region/multi-data centers 09:01:43 I hope fernet token can work in multi site scenario 09:01:55 I am sure it can 09:02:15 so I will send an email about the keystonemiddleware 09:02:36 some bugs may need to be fixed. 09:02:40 ok, please 09:03:35 #action: mail about keystonemiddleware in openstack-dev 09:04:01 thanks. it's time to wrap up the meeting 09:04:52 Thank you attend the meeting. see you 09:05:17 #endmeeting