07:09:00 #startmeeting multisite 07:09:00 Meeting started Thu Sep 22 07:09:00 2016 UTC. The chair is joehuang. Information about MeetBot at http://wiki.debian.org/MeetBot. 07:09:00 Useful Commands: #action #agreed #help #info #idea #link #topic. 07:09:00 The meeting name has been set to 'multisite' 07:09:08 hi 07:09:11 hi 07:09:18 #topic rollcall 07:09:23 #info joehuang 07:09:30 #info Ashish 07:10:34 #info dimitri 07:10:51 #topic Resources Sync 07:11:12 we listed resource sync like: 07:11:21 3.1)SEG 07:11:21 3.2)FLAVOR 07:11:22 3.3)SSH KEY 07:11:48 let's talk about these items 07:12:04 and other Revisit Geographical Redundancy 07:12:05 4)cross neutron networking automation (joehuang, 07:28:31) 07:12:05 5) installation cooperation (joehuang, 07:32:18) 07:12:09 if needed 07:12:09 Before that, can we talk about the unstable lab 07:12:22 lab is a problem 07:12:45 neutron ovs agent is malfunctioning 07:13:18 and all the newly created networks are having problems 07:13:20 is it possible dynamically create an SUT? 07:13:37 that is what we thought initially 07:14:05 not unless we automate m-r creation 07:14:13 fuel has to allow MR 07:14:26 yes, otherwise the environment is very easy to be unstable 07:15:18 i can ask to reinstall it. but that may work until something else goes wrong 07:16:03 how about just use devstack multi-region as our SUT? 07:16:28 devstack support multi-region setup 07:16:57 zhiyuan has fixed one issue in the script, if they did not ruin it after then, it should works 07:17:01 then what’s the point of the lab? 07:17:21 the current problem has started occuring from the time there was a shutdown of the nodes 07:17:55 i still believe it can be i 07:17:57 @joe: You have shanghai lab right? 07:17:57 SAshish: Error: "joe:" is not a valid command. 07:18:07 fixed without reinstall 07:18:31 it is based on which installer? 07:18:45 shanghai lab based on compass4nfv 07:18:58 is it stable since then? 07:19:03 but it also doesn't support MR automation 07:19:19 manaully setup 07:19:40 during berlin summit, functest and CI talked about to setup multisite lab 07:19:48 but it take time 07:20:25 we will not change the lab now 07:20:44 was just asking if we can change the installer 07:21:02 no installer supports m-r 07:21:02 If that is having MR/Stable 07:21:07 non installer support multi-region automation very well 07:21:26 to Dimitri, +1 07:21:28 fuel is the best optiom ,because it can dynamically create environments at least 07:21:37 so there’s a sort of m-r setup 07:21:44 just not with centralized keystone 07:21:57 multiple controllers 07:22:52 I will see what can be done with the lab 07:23:05 ok 07:23:31 I was able to ping the instances which were created on a network after few configurations at ovs level directly 07:23:52 but the problem comes when we create new network 07:24:11 And after the patches sync-ed to colorado branch, the tagging for colorado.1.0 in multisite repo finished 07:25:18 so can we assume that colorado is done? 07:25:50 at least for colorado.1.0 done, the release manager planned 1.0, 2.0, 3.0 07:26:01 good 07:26:09 nice, congrats guys 07:26:10 :) 07:26:10 we can focus on the trunk for next D 07:26:29 :) 07:26:35 yes 07:27:00 enjoy the moment with something delivered 07:27:36 the lab can be fixed the other day 07:28:33 so try to fix it without reinstallation first 07:29:12 if some issues come again, then let's figure out what's least cost way to maintain the lab 07:29:38 last time it was memcache and keystone 07:30:20 now it is neutron, I will take help from a friend here and try to fix it 07:31:05 great, Ashish 07:31:56 also consult the progress of multi-site infra in functest 07:32:16 so, let's come back to the feature topic? 07:33:05 sure 07:33:52 last time, sec topic was raised, and mail-thread about it 07:34:53 I agreed with Ashish 07:35:00 "We must use neutron to perform all our operations as with neutron we have total control over it" 07:35:45 yeah, and one more thing is 07:35:46 I think sync itself (if excluding the remote sec-group) is not complex, the complexity is to ensure the rules set in different region of Neutron will not conflict with each other. Otherwise, it'll become mess. 07:36:08 from this how we make sure there is no conflict 07:37:23 so, If there is a sec group s1 in r1, we take all the rules of s1 and check if there are any sec groups in r2 which has all the rules as s1 07:37:26 the questions that need answer are: how the API will look like? will we allow granular sync? will the use trigger sync manually? what should be the auto-sync criteria 07:38:18 will this construct make sense: 07:38:45 manual and auto-sync both, when ever a user creates a sec group in reg 1, he can use manual sync to sync it to other regions 07:38:56 kingbird sync —resource -region Region1 —region Region2 07:40:10 how to address the conflict? 07:40:19 automated can be achieved by e.g. a cron trigger - a workflow in Mistral 07:40:42 if you get an error from a certain region - show that sync faild for that resource in that region 07:41:00 so we may allow partial sync as well 07:41:09 call it best-effor 07:41:10 t 07:41:10 partial == not in all regions?? 07:41:13 yes 07:41:38 we can of course have a rollback mechanism for each sync 07:41:52 that can be a separate option 07:42:08 how can we rollback? 07:42:31 if at least one error occured - iterate through the chosen region and delete the added resource 07:42:39 I mean once is sync happens there is no way where it can tell that for this region these are the original sec groups 07:42:43 chosen regions* 07:42:52 yes you can 07:43:09 the uuid in different region is different 07:43:11 the targer regions are your input 07:43:15 okay, during sync process itself we have that info 07:43:18 how about the next time sync? 07:43:33 the next time sync is a separate operation 07:43:42 if it fails, it has no implication to the previos sync 07:43:45 conflict is not an issue 07:43:56 I mean "we can of course have a rollback mechanism for each sync" 07:43:56 plus the names are unique 07:43:58 in this context 07:44:08 if you try to add the same resource, with the same name, it’ll fail 07:44:11 just copy all from one region to another? 07:44:21 ok 07:44:24 consider this 07:44:28 sync reg1 and reg 2 07:44:30 I’m in region one 07:44:34 means reg <> reg 07:44:47 I wan to sync a flavor to region4 and region5 07:44:57 I may wanna do this 07:45:44 kingbird sync -type flavor —name —region region4 —region region5 07:46:19 questions? 07:46:25 one user sync reg1->reg2, and if another user of the same tenant issue sync reg2->1, what will happened? 07:46:39 reg1, reg4, reg5 all will be synced 07:47:00 concurrency guarantee 07:47:05 joe, depends on who succeeds first 07:47:18 but let’s wait with those exceptional cases 07:47:18 it'll become mess 07:48:03 joe, why will it be a mess 07:49:12 so, which region is responsible for security rule creation, one or many? 07:49:52 if we want to roll back on failure, we can have the following construct kingbird sync -type flavor —name —region region4 —region region5 —rollback 07:50:17 this will ensure, that if at least one region sync fails, the newly added resources to other regions will be deleted 07:50:25 rollback has to happen how heat stack update happens 07:50:29 yes 07:50:32 during sync itself 07:50:41 not a separate triggering 07:50:47 during sync of course 07:50:50 one question... After sync is completed: flavor(R1/R4/R5) = flavor(R1 + R4 + R5) OR just flavor R1 is copied to R4 and R5 07:51:20 yes, it’s simply copied 07:51:22 so it is same from region 4: kingbird sync -type flavor —name —region region1 —region region5 07:51:31 yes! 07:51:55 then all the regions are in sync w.r.t flavor at the end 07:51:57 the region you copy from is first, the region where kb is installed 07:51:59 so each region is allowed to be updated? 07:52:21 it’s up to the user to choose the regions for sync 07:52:28 I might have 3 regions 07:52:34 but only want to sync flavors in two 07:52:36 all regions here means these three regions 07:52:43 okay 07:53:05 then there will a sync for all the regions 07:53:34 can be like this --all-regions 07:54:55 perhaps, but that is also an addition 07:55:27 which region is responsible for CRUD of flavor? or each region? 07:55:44 whichever has kingbird installed 07:56:07 KB will login to diff regions and prefrom CRUD 07:56:12 potentially all regions can have kb installed, but in that case we should create a mechanism that disables quota sync 07:56:25 so that the regions don’t compete 07:56:41 centrally we should manage 07:56:42 concurrency is an issue 07:57:02 what is the issue with centralized KB ? 07:57:10 as we have now with QM 07:57:19 none 07:57:26 it’s actually ok 07:57:27 or 07:57:54 it just makes the region it’s installed in - a default one 07:58:30 kingbird sync —type flavor —name —from region1 —to region2 —to region3 07:58:33 so the source region should be fixed? 07:58:34 what about this 07:58:47 with the above interface not necessarily 07:59:33 region is dynamic 07:59:58 you mean source region could be flexible? 08:00:40 yes 08:00:42 time is up 08:00:45 we can even have the —force option 08:00:48 wait :) 08:01:01 the force option will overwrite a resource if it exists in a certain region 08:01:14 so of I have this flavor in region3 08:01:27 conflict detection will be hectic 08:01:34 with the force option kb will delete this flavor and create a new copy 08:01:41 if conflict overwrite it 08:01:56 for now we can use resource names as unique identifiers 08:02:24 some resources name is not unique key 08:02:28 no Dimtri, same rules are be there with other names 08:02:43 same configurations can be there with other flavors 08:02:52 are => can 08:03:09 yes 08:03:14 that’s correct 08:03:14 resource name doesnt look apt 08:03:29 that is the reason I told one thing in mailing list. 08:03:34 ok, let’s not rush with force overwrite 08:03:51 not even for force overwrite 08:04:08 even for creating a flavor/SEG as well 08:04:37 if S1 has rule 22 open in R1, and in R2 it is called G1 but the rule is also 22 open 08:04:51 Based on this and your approach can you start a google doc where we can gradually add info? 08:04:52 so we should not copy S1 to R2 08:05:06 you can also create a blueprint and attach the doc to it 08:05:52 let's work offline, and continue the disucssion in m-l and next meeting, I have to go now 08:05:58 sure 08:06:04 ok 08:06:08 thanks everyone 08:06:11 #endmeeting