07:09:00 <joehuang> #startmeeting multisite
07:09:00 <collabot> Meeting started Thu Sep 22 07:09:00 2016 UTC.  The chair is joehuang. Information about MeetBot at http://wiki.debian.org/MeetBot.
07:09:00 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
07:09:00 <collabot> The meeting name has been set to 'multisite'
07:09:08 <sorantis> hi
07:09:11 <joehuang> hi
07:09:18 <joehuang> #topic rollcall
07:09:23 <joehuang> #info joehuang
07:09:30 <SAshish> #info Ashish
07:10:34 <sorantis> #info dimitri
07:10:51 <joehuang> #topic Resources Sync
07:11:12 <joehuang> we listed resource sync like:
07:11:21 <joehuang> 3.1)SEG
07:11:21 <joehuang> 3.2)FLAVOR
07:11:22 <joehuang> 3.3)SSH KEY
07:11:48 <joehuang> let's talk about these items
07:12:04 <joehuang> and other Revisit Geographical Redundancy
07:12:05 <joehuang> 4)cross neutron networking automation (joehuang, 07:28:31)
07:12:05 <joehuang> 5) installation cooperation (joehuang, 07:32:18)
07:12:09 <joehuang> if needed
07:12:09 <SAshish> Before that, can we talk about the unstable lab
07:12:22 <joehuang> lab is a problem
07:12:45 <SAshish> neutron ovs agent is malfunctioning
07:13:18 <SAshish> and all the newly created networks are having problems
07:13:20 <joehuang> is it possible dynamically create an SUT?
07:13:37 <SAshish> that is what we thought initially
07:14:05 <sorantis> not unless we automate m-r creation
07:14:13 <SAshish> fuel has to allow MR
07:14:26 <joehuang> yes, otherwise the environment is very easy to be unstable
07:15:18 <sorantis> i can ask to reinstall it. but that may work until something else goes wrong
07:16:03 <joehuang> how about just use devstack multi-region as our SUT?
07:16:28 <joehuang> devstack support multi-region setup
07:16:57 <joehuang> zhiyuan has fixed one issue in the script, if they did not ruin it after then, it should works
07:17:01 <sorantis> then what’s the point of the lab?
07:17:21 <SAshish> the current problem has started occuring from the time there was a shutdown of the nodes
07:17:55 <sorantis> i still believe it can be i
07:17:57 <SAshish> @joe: You have shanghai lab right?
07:17:57 <collabot> SAshish: Error: "joe:" is not a valid command.
07:18:07 <sorantis> fixed without reinstall
07:18:31 <SAshish> it is based on which installer?
07:18:45 <joehuang> shanghai lab based on compass4nfv
07:18:58 <SAshish> is it stable since then?
07:19:03 <joehuang> but it also doesn't support MR automation
07:19:19 <joehuang> manaully setup
07:19:40 <joehuang> during berlin summit, functest and CI talked about to setup multisite lab
07:19:48 <joehuang> but it take time
07:20:25 <sorantis> we will not change the lab now
07:20:44 <SAshish> was just asking if we can change the installer
07:21:02 <sorantis> no installer supports m-r
07:21:02 <SAshish> If that is having MR/Stable
07:21:07 <joehuang> non installer support multi-region automation very well
07:21:26 <joehuang> to Dimitri, +1
07:21:28 <sorantis> fuel is the best optiom ,because it can dynamically create environments at least
07:21:37 <sorantis> so there’s a sort of m-r setup
07:21:44 <sorantis> just not with centralized keystone
07:21:57 <SAshish> multiple controllers
07:22:52 <sorantis> I will see what can be done with the lab
07:23:05 <joehuang> ok
07:23:31 <SAshish> I was able to ping the instances which were created on a network after few configurations at ovs level directly
07:23:52 <SAshish> but the problem comes when we create new network
07:24:11 <joehuang> And after the patches sync-ed to colorado branch, the tagging for colorado.1.0 in multisite repo finished
07:25:18 <sorantis> so can we assume that colorado is done?
07:25:50 <joehuang> at least for colorado.1.0 done, the release manager planned 1.0, 2.0, 3.0
07:26:01 <sorantis> good
07:26:09 <SAshish> nice, congrats guys
07:26:10 <SAshish> :)
07:26:10 <joehuang> we can focus on the trunk for next D
07:26:29 <joehuang> :)
07:26:35 <SAshish> yes
07:27:00 <joehuang> enjoy the moment with something delivered
07:27:36 <joehuang> the lab can be fixed the other day
07:28:33 <joehuang> so try to fix it without reinstallation first
07:29:12 <joehuang> if some issues come again, then let's figure out what's least cost way to maintain the lab
07:29:38 <SAshish> last time it was memcache and keystone
07:30:20 <SAshish> now it is neutron, I will take help from a friend here and try to fix it
07:31:05 <joehuang> great, Ashish
07:31:56 <joehuang> also consult the progress of multi-site infra in functest
07:32:16 <joehuang> so, let's come back to the feature topic?
07:33:05 <SAshish> sure
07:33:52 <joehuang> last time, sec topic was raised, and mail-thread about it
07:34:53 <joehuang> I agreed with Ashish
07:35:00 <joehuang> "We must use neutron to perform all our operations as with neutron we have total control over it"
07:35:45 <SAshish> yeah, and one more thing is
07:35:46 <SAshish> I think sync itself (if excluding the remote sec-group) is not complex, the complexity is to ensure the rules set in different region of Neutron will not conflict with each other. Otherwise, it'll become mess.
07:36:08 <SAshish> from this how we make sure there is no conflict
07:37:23 <SAshish> so, If there is a sec group s1 in r1, we take all the rules of s1 and check if there are any sec groups in r2 which has all the rules as s1
07:37:26 <sorantis> the questions that need answer are: how the API will look like? will we allow granular sync? will the use trigger sync manually? what should be the auto-sync criteria
07:38:18 <sorantis> will this construct make sense:
07:38:45 <SAshish> manual and auto-sync both, when ever a user creates a sec group in reg 1, he can use manual sync to sync it to other regions
07:38:56 <sorantis> kingbird sync —resource <uuid> -region Region1 —region Region2
07:40:10 <joehuang> how to address the conflict?
07:40:19 <sorantis> automated can be achieved by e.g. a cron trigger - a workflow in Mistral
07:40:42 <sorantis> if you get an error from a certain region - show that sync faild for that resource in that region
07:41:00 <sorantis> so we may allow partial sync as well
07:41:09 <sorantis> call it best-effor
07:41:10 <sorantis> t
07:41:10 <SAshish> partial == not in all regions??
07:41:13 <sorantis> yes
07:41:38 <sorantis> we can of course have a rollback mechanism for each sync
07:41:52 <sorantis> that can be a separate option
07:42:08 <SAshish> how can we rollback?
07:42:31 <sorantis> if at least one error occured - iterate through the chosen region and delete the added resource
07:42:39 <SAshish> I mean once is sync happens there is no way where it can tell that for this region these are the original sec groups
07:42:43 <sorantis> chosen regions*
07:42:52 <sorantis> yes you can
07:43:09 <joehuang> the uuid in different region is different
07:43:11 <sorantis> the targer regions are your input
07:43:15 <SAshish> okay, during sync process itself we have that info
07:43:18 <joehuang> how about the next time sync?
07:43:33 <sorantis> the next time sync is a separate operation
07:43:42 <sorantis> if it fails, it has no implication to the previos sync
07:43:45 <SAshish> conflict is not an issue
07:43:56 <SAshish> I mean "we can of course have a rollback mechanism for each sync"
07:43:56 <sorantis> plus the names are unique
07:43:58 <SAshish> in this context
07:44:08 <sorantis> if you try to add the same resource, with the same name, it’ll fail
07:44:11 <joehuang> just copy all from one region to another?
07:44:21 <sorantis> ok
07:44:24 <sorantis> consider this
07:44:28 <SAshish> sync reg1 and reg 2
07:44:30 <sorantis> I’m in region one
07:44:34 <SAshish> means reg <> reg
07:44:47 <sorantis> I wan to sync a flavor to region4 and region5
07:44:57 <sorantis> I may wanna do this
07:45:44 <sorantis> kingbird sync -type flavor —name <name_or_uuid> —region region4 —region region5
07:46:19 <sorantis> questions?
07:46:25 <joehuang> one user sync reg1->reg2, and if another user of the same tenant issue sync reg2->1, what will happened?
07:46:39 <SAshish> reg1, reg4, reg5 all will be synced
07:47:00 <joehuang> concurrency guarantee
07:47:05 <sorantis> joe, depends on who succeeds first
07:47:18 <sorantis> but let’s wait with those exceptional cases
07:47:18 <joehuang> it'll become mess
07:48:03 <sorantis> joe, why will it be a mess
07:49:12 <joehuang> so, which region is responsible for security rule creation, one or many?
07:49:52 <sorantis> if we want to roll back on failure, we can have the following construct kingbird sync -type flavor —name <name_or_uuid> —region region4 —region region5 —rollback
07:50:17 <sorantis> this will ensure, that if at least one region sync fails, the newly added resources to other regions will be deleted
07:50:25 <SAshish> rollback has to happen how heat stack update happens
07:50:29 <sorantis> yes
07:50:32 <SAshish> during sync itself
07:50:41 <SAshish> not a separate triggering
07:50:47 <sorantis> during sync of course
07:50:50 <SAshish> one question... After sync is completed: flavor(R1/R4/R5) = flavor(R1 + R4 + R5)  OR just flavor  R1 is copied to R4 and R5
07:51:20 <sorantis> yes, it’s simply copied
07:51:22 <SAshish> so it is same from region 4: kingbird sync -type flavor —name <name_or_uuid> —region region1 —region region5
07:51:31 <sorantis> yes!
07:51:55 <SAshish> then all the regions are in sync w.r.t flavor at the end
07:51:57 <sorantis> the region you copy from is first, the region where kb is installed
07:51:59 <joehuang> so each region is allowed to be updated?
07:52:21 <sorantis> it’s up to the user to choose the regions for sync
07:52:28 <sorantis> I might have 3 regions
07:52:34 <sorantis> but only want to sync flavors in two
07:52:36 <SAshish> all regions here means these three regions
07:52:43 <SAshish> okay
07:53:05 <SAshish> then there will a sync for all the regions
07:53:34 <SAshish> can be like this --all-regions
07:54:55 <sorantis> perhaps, but that is also an addition
07:55:27 <joehuang> which region is responsible for CRUD of flavor? or each region?
07:55:44 <sorantis> whichever has kingbird installed
07:56:07 <SAshish> KB will login to diff regions and prefrom CRUD
07:56:12 <sorantis> potentially all regions can have kb installed, but in that case we should create a mechanism that disables quota sync
07:56:25 <sorantis> so that the regions don’t compete
07:56:41 <SAshish> centrally we should manage
07:56:42 <joehuang> concurrency is an issue
07:57:02 <SAshish> what is the issue with centralized KB ?
07:57:10 <SAshish> as we have now with QM
07:57:19 <sorantis> none
07:57:26 <sorantis> it’s actually ok
07:57:27 <sorantis> or
07:57:54 <sorantis> it just makes the region it’s installed in - a default one
07:58:30 <sorantis> kingbird sync —type flavor —name <name> —from region1 —to region2 —to region3
07:58:33 <joehuang> so the source region should be fixed?
07:58:34 <sorantis> what about this
07:58:47 <sorantis> with the above interface not necessarily
07:59:33 <SAshish> region is dynamic
07:59:58 <joehuang> you mean source region could be flexible?
08:00:40 <sorantis> yes
08:00:42 <joehuang> time is up
08:00:45 <sorantis> we can even have the —force option
08:00:48 <sorantis> wait :)
08:01:01 <sorantis> the force option will overwrite a resource if it exists in a certain region
08:01:14 <sorantis> so of I have this flavor in region3
08:01:27 <SAshish> conflict detection will be hectic
08:01:34 <sorantis> with the force option kb will delete this flavor and create a new copy
08:01:41 <SAshish> if conflict overwrite it
08:01:56 <sorantis> for now we can use resource names as unique identifiers
08:02:24 <joehuang> some resources name is not unique key
08:02:28 <SAshish> no Dimtri, same rules are be there with other names
08:02:43 <SAshish> same configurations can be there with other flavors
08:02:52 <SAshish> are => can
08:03:09 <sorantis> yes
08:03:14 <sorantis> that’s correct
08:03:14 <SAshish> resource name doesnt look apt
08:03:29 <SAshish> that is the reason I told one thing in mailing list.
08:03:34 <sorantis> ok, let’s not rush with force overwrite
08:03:51 <SAshish> not even for force overwrite
08:04:08 <SAshish> even for creating a flavor/SEG as well
08:04:37 <SAshish> if S1 has rule 22 open in R1, and in R2 it is called G1 but the rule is also 22 open
08:04:51 <sorantis> Based on this and your approach can you start a google doc where we can gradually add info?
08:04:52 <SAshish> so we should not copy S1 to R2
08:05:06 <sorantis> you can also create a blueprint and attach the doc to it
08:05:52 <joehuang> let's work offline, and continue the disucssion in m-l and next meeting, I have to go now
08:05:58 <SAshish> sure
08:06:04 <sorantis> ok
08:06:08 <sorantis> thanks everyone
08:06:11 <joehuang> #endmeeting