08:12:34 <joehuang> #startmeeting multisite
08:12:34 <collabot> Meeting started Thu Aug 20 08:12:34 2015 UTC.  The chair is joehuang. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:12:34 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:12:34 <collabot> The meeting name has been set to 'multisite'
08:12:46 <joehuang> #topic rollcall
08:12:52 <joehuang> #info joehuang
08:13:22 <Malla> #info Malla
08:13:25 <hafe> #info hafe
08:14:06 <joehuang> #topic usecase 1 verfication
08:14:13 <sorantis> #info Dimitri
08:14:27 <joehuang> hello, a short summary of the verification
08:14:38 <joehuang> the test is started by hafe
08:15:10 <joehuang> and the first step is to verify the cluster async. replication
08:16:03 <joehuang> the DB aync. replication between cluster works
08:16:28 <hafe> I also got master replication between galera cluster to work
08:16:38 <joehuang> but hafe found that the token validation request was sent to the first keystone always
08:16:40 <hafe> clusters
08:16:56 <hafe> yeah seems to be a keystonemiddleware issue
08:17:12 <joehuang> i debuged into the source code
08:17:27 <joehuang> found that there is some issues in the source code
08:18:26 <joehuang> But even when I make the keystonemiddleware send the token validation to the right keystone server
08:19:02 <joehuang> the second keystone server did not validate the token correctly
08:19:54 <joehuang> I did not have a lot of time to find out why, but I found that the token validation is sent to the http://keyston IP: 5000/
08:20:14 <hafe> which is wrong, should be sent to the admin port
08:20:18 <hafe> not the public one
08:20:19 <MatthewLi> #info Jun Li
08:20:22 <joehuang> yes
08:20:26 <joehuang> correct
08:20:36 <hafe> but I tested and it does not help
08:20:57 <joehuang> in my own environment which is used to correct the bug in keystonemiddleware
08:21:11 <hafe> should send an email to openstack(-dev) and ask
08:21:18 <joehuang> the token validation request is sent to http://keyston IP: 35357/
08:22:02 <joehuang> that works in my own enviroment. will send a mail after more investigation
08:22:29 <hafe> ok great
08:22:54 <hafe> I am sure this can be fixed, can we look at it from an arch point of view?
08:23:25 <joehuang> And from the verification, I proposal to use a star mode " a cluster " with lot of "async slaves"
08:23:39 <joehuang> to hafe: agree
08:23:57 <hafe> just to be clear;
08:24:19 <hafe> how keystone is deployed and how the database is replicated is just characteristics
08:24:31 <hafe> no (?) functional impact
08:24:33 <joehuang> KeyStone server with the cluster of DB will be deployed in two sites
08:25:15 <hafe> correct?
08:25:24 <joehuang> for site level redundancy to provide write capability
08:26:11 <joehuang> and all other slaves which is async-replicated with DB will work as read-only keystone server
08:26:41 <joehuang> and in other sites, two or more slaves could be deployed for redundancy purpose
08:28:59 <joehuang> for functional impact, slave sites will be only used for token validation
08:29:59 <joehuang> but for create/update/delete function on KeyStone, should be done on the master sites: the two sites with DB cluster.
08:30:23 <joehuang> Is it more clear now?
08:32:18 <joehuang> Hi, Hafe, you mentioned write/read split in DB? could you describe in more detail?
08:33:09 <hafe> well I am no database expert but clients can in scale out solution have different endpoints for write and read
08:33:30 <hafe> client in this case is keystone
08:33:42 <hafe> it has a single DB endpoint
08:34:18 <hafe> so keystone in "slave" region cannot mutate any data
08:34:52 <hafe> that restriction could be lifted with a write/read split
08:35:38 <joehuang> will the keystone in "slave" region access the db in local site or remote site?
08:35:52 <hafe> local site
08:36:08 <hafe> during token validation there will be db reads that you want to be local
08:36:43 <joehuang> yes
08:37:11 <hafe> but again all this is just characteristics, no function impact
08:37:29 <hafe> you can have a central keystone server that all sites use
08:37:37 <hafe> and deal with faults with that
08:38:35 <joehuang> the db configuration in slave region keystones can be configured separately
08:39:11 <hafe> what do you mean?
08:40:32 <joehuang> "you can have a central keystone server that all sites use" what does the central keystone server stand for
08:41:46 <hafe> one keystone installation at a central site
08:42:21 <joehuang> how to disable the write in the "slave" region, through API policy or at DB level
08:43:24 <hafe> right now DB level
08:43:53 <hafe> the keystone DB user should probably not have write access
08:43:59 <hafe> on slaves
08:44:14 <joehuang> hafe, not follow your idea for the central keystone. What's the main purpose for the central installation
08:44:19 <hafe> just to hinder accidental writes by some admin or so
08:45:09 <hafe> what I try to say is that how keystone and its db is deployed only affects characteristics
08:45:21 <hafe> not functional impact really
08:45:47 <hafe> you could have a single keystone server instance running in China serving all regions
08:45:58 <hafe> still works until some links goes down
08:45:58 <joehuang> ok. understand, no new feature required
08:46:18 <hafe> we are just looking at improving characteristics
08:46:31 <hafe> in terms of keystone API latency and HA
08:46:40 <hafe> site level failures
08:47:42 <MatthewLi> not even see one line code in the multisite git repo, why so hot discussion here
08:47:59 <joehuang> agree, for keystone, multisite project focus on how to deploy keystone
08:48:20 <joehuang> we don't need to put too much effort on it
08:48:23 <hafe> no
08:48:43 <hafe> the prototype was just an idea on how it can be done by the installer which is outside our scope
08:48:52 <joehuang> yes
08:49:11 <hafe> but what about ETSI NFV and this matter?
08:49:21 <hafe> are there any related reqs?
08:51:24 <joehuang> the prototype is to verify that keystone can work in multisite scenario with site level reliability, If don't, new feature required in keystone.
08:52:12 <joehuang> when we prioritized the use case, most of people put the identity in the 1st priority.
08:52:54 <hafe> so site level reliability implies something like in the prototype?
08:53:02 <joehuang> yes
08:53:18 <joehuang> if each site with its own keystone server
08:53:37 <joehuang> then even this site failure, other site can still work with identity service
08:54:19 <joehuang> if fernet works. we have nothing to do in indentity management for multisite scenario
08:54:19 <hafe> well it has more todo with management/orchestration I guess?
08:55:35 <joehuang> management/orchestration still rely on keystone for openstack identity managment and token validation
08:55:50 <joehuang> especially token validation
08:57:26 <joehuang> if the site where keystone server installed for the token validation is failed, and other sites will not able to handle all requests which use the keystone server for token validation
08:58:51 <hafe> multi site == multi region cloud ?
08:59:16 <hafe> single cloud, multiple region
08:59:17 <joehuang> currently all keystone deployment possibility can not guarranty site level keystone avaialbiltiy except fernet token based keystone installed in each site
08:59:46 <joehuang> single cloud, multi-region/multi-data centers
09:01:43 <joehuang> I hope fernet token can work in multi site scenario
09:01:55 <hafe> I am sure it can
09:02:15 <hafe> so I will send an email about the keystonemiddleware
09:02:36 <joehuang> some bugs may need to be fixed.
09:02:40 <joehuang> ok, please
09:03:35 <joehuang> #action: mail about keystonemiddleware in openstack-dev
09:04:01 <joehuang> thanks. it's time to wrap up the meeting
09:04:52 <joehuang> Thank you attend the meeting. see you
09:05:17 <joehuang> #endmeeting