14:58:10 #startmeeting OPNFV Pharos 14:58:10 Meeting started Wed Sep 2 14:58:10 2015 UTC. The chair is trevor_intel. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:58:10 Useful Commands: #action #agreed #help #info #idea #link #topic. 14:58:10 The meeting name has been set to 'opnfv_pharos' 14:58:18 #info trevor_intel 14:58:38 #info iMSWynne 14:58:47 #info Narinder Gupta 14:58:57 #info Chris Price 14:58:59 #info Fatih Degirmenci 15:00:19 #info Proposal for agenda topics ... 1) MAAS PoC 2) Rls B Planning (includes CI usage of labs, phaors spec, tooling topics) 3) Lab mangement (but unlikely to get there today) 15:00:44 #info Comments / suggestions on agenda please 15:01:33 Lab requirements? Might be part of b rls planning 15:02:34 Yes lets list out all rls B topics when we get there, ok? 15:02:56 IRC only? 15:03:01 yes 15:03:07 Ok for me. :) 15:03:13 trevor_intel: ok for me too 15:03:36 ok if we struggle I will setup a bridge for future 15:03:47 #topic MAAS PoC 15:03:47 Good not to 15:03:57 trevor_intel: :) 15:04:03 i can give us a webex if you want to have voice and recording? 15:04:21 lets try IRC for now 15:04:34 hello 15:04:34 thanks Iben 15:04:43 #info Iben Rodriguez 15:05:05 everyone - please info in so we know who you are... 15:05:12 trevor_intel: https://wiki.opnfv.org/maas_requirement is the maas requirement page which i put together to setup MAAS in LF lab so that it can assign the community lab for deployment 15:05:17 iben: we already did that 15:05:24 sorry I’m late - missed it 15:05:25 #info daniel smith 15:05:26 #Info MAAS team did a demo, made a video and wrote up some requirements 15:05:53 #info artur tyloch 15:05:55 #info Desire is to do a POC using a community lab and LF infra 15:06:25 #info A plan to execute is needed 15:06:30 so if I want to use MaaS in my community lab it can be “driven” by the central controller already in LF? 15:07:06 There is a regional controller in LF lab 15:07:08 iben: i do not have idea about central controller. is it something like jumpstart? 15:07:45 and the cluster controller is what I setup in my remote lab - outside the LF infra - right? 15:08:21 iben: correct 15:08:27 so communication between regional controller and cluster controller take place over internet - wide area network? 15:08:35 Yes ... lets focus on planning for the PoC 15:08:37 have some questions regarding inclusion of LF into POC 15:08:44 that sounds great - let’s do it. 15:08:46 iben: basically cluster controller controls the dns/dhcp/tftp on that local cluster of hardware 15:09:17 fdegir: questions? 15:09:19 #info do we need to reserve hw in LF dedicated to MAAS? 15:09:41 #info Sicne we already lack hw so it won't be good to lose yet another one 15:10:11 fdegir: if possible we can live with a VM provided we have bridges on the existing controller server. 15:10:22 both the regional controller and the cluster controller can be virtual - no? 15:10:44 iben: yes it can be as far as it has bridges to admin and ipmi network 15:10:49 our plan is to use 6 servers coming from POD1 for builds, virt deploys etc. 15:11:06 if MAAS VM can be colocated with one of the other purposed servers, like build server 15:11:07 but region controller won’t ipmi - that is the job of the cluster controller inthe remote lab 15:11:11 than it could be OK 15:11:15 iben: correct 15:11:19 Isolating the MaaS on LF & remote would be good. Tunneled VM's if possible. 15:11:48 is there a desire for 2 region controllers for HA ? 15:11:49 ChrisPriceAB: cluster controller do the rpc calls to regional controller 15:12:13 uh.. Tunneled' VM's through MAAS? that doesnt make sense to me 15:12:33 MAAS will deploy a OStack Infrastructure and then youw ould have to use OStack to do a chaining between the two networks 15:12:48 MAAS will only deploy your enviornment.. not have view of the VMs under OStack control. 15:12:48 lmcdasm: agreed 15:13:15 lmcdasm: maas just a deployment tool not a monitoring service tool 15:13:24 Running MaaS in VM's 15:14:37 ? 15:14:38 another question: would MAAS controlled LF POD require changes on installers? 15:15:02 MasS if being discussed here only to controll remote labs now 15:15:09 what is the recommended resource allocation for a cluster controller MAAS VM? 15:15:10 Chris - you want to have multple MAAS VM's that deploy to hardware talking to each other? 15:15:21 there isn’t hardware in LF to control yet - is there? 15:15:25 fdegir: MAAS can install the installers on bare metal or VM and then handover to different installers to move forward 15:15:59 iben: the subject of the discussion is 15:16:02 Desire is to do a POC using a community lab and LF infra 15:16:13 so that's why I question inclusion of LF into MAAS POC 15:16:21 for community labs, I'm fine 15:16:51 understood - jsut trying to clarrify - we are talking (I think) about a few vms (just 1?) in the lf lab controlling remote systems in community labs 15:17:34 and that was my other question; if MAAS VM can be colocated on a server located in LF 15:17:36 sorry - what is the benefit of having a Controller in LF that does deployment over a VPN tunnel to Metal Machines in other Dcs? 15:17:39 without costing us the entire server 15:17:43 than I'm fine with that as well 15:17:46 i dont see the value in the use case - maybe i dont understand the case 15:17:47 from the drawing - https://wiki.opnfv.org/maas_requirement - there is a HA for the region controller in the LF lab - wanted to know if that needed 2 VMs 15:18:09 given that the LF POD is not included into POC in the beginning 15:18:30 iben: HA needed mutiple vms yes for clustering 15:18:35 it’s like having jenkins central infra in lf lab - tho control slaves in remote labs 15:18:47 is that right MaaS experts? 15:19:16 well.. except that jenkins can call mutlple deployrs (the cluster controllers at local site in he digram) 15:19:23 whereas MAAS will only deploy that one cluster type 15:20:03 so you see a chain where JJB, when its a JOID test CI run, will call the reginal controller that will hook to a communicty lab and do a buikd, merge, verify, etc? 15:20:52 I'm not sure how Jenkins can fit into this picture 15:20:56 me neither - 15:21:00 we can request resources from MAAS 15:21:05 but maybe for the deployment poc it doesnt matter 15:21:11 by a job 15:21:14 since MAAS is the same FUEL essentiall 15:21:18 and once the resource becomes available 15:21:22 they are both metal deployers 15:21:23 we do the rest of the stuff 15:21:28 which might complicate things a bit 15:21:33 so this POC is to hooku p the chain between the Over Cluster and local cluster 15:21:35 which is fine 15:21:42 and 15:21:46 I want to ask another question 15:21:46 but im trying to see the case / afterwards - otehr than just a higher level of control 15:21:51 what is the purpose of the POC? 15:22:05 is it making resources available for CI/automation? 15:22:17 or making resources managed for community use/development purposes? 15:22:23 or both? 15:22:36 or deploying fuel/foreman through MAAS 15:22:43 fair questino - what is the "proof point" here 15:22:57 for the Proof of Concept - that we can do a chained deploytment through two controllers 15:22:58 ? 15:23:12 Is it not to make community labs multipurpose? 15:23:22 hmm. ok - 15:23:27 what do we mean multipurpose a 15:23:33 and how does this poc satify that? 15:23:44 fdegir: lmcdasm POC was for automate the deployment once release is available on community lab through LF lab 15:23:58 narindergupta: deployment of what? 15:24:04 im wondering of what - 15:24:05 opnfv? 15:24:12 fdegir: depoloyment of didferent installers 15:24:15 since FOREMAN and FUEL already are able to deploy 15:24:30 and i see this as something you would want to run under those deployers? 15:24:40 so this will essentially add another layer between jenkins and the deployment mechanism 15:24:55 which will require changes in CI definitely 15:24:59 and perhaps in installers 15:24:59 so you would have MAAS calling a FUEL install to deploy OSTACK (you have a second VM you dont need then). 15:25:02 yup! 15:25:09 jenkins -> MAAS -> fuel/foreman 15:25:18 hmm. 15:25:25 is this the use case? 15:25:33 fdegir: correct jenkins -> MAAS -> fuel/foreman/JOID 15:25:51 can we do other way 15:25:54 there are two huge benefits with MaaS in LF (1) - we get centralized visibility for build out in community labs - yet they remain isolated - good for security and such - (2) - we get fast multi-os bare metal and virtual OS installation with all the juju charms and other such giving us rapid provisioing for OpenDaylight and other fun things we want 15:25:58 jenkins requests resource fro MAAS 15:26:06 oh yeah - IPv6 is supported too 15:26:09 and once we get the resources, jenkins does the deploy 15:26:36 also - i thoguht we were getting hardware for LF lab and would use the MaaS for that 15:26:39 since I thought the first benefit we can get from MAAS is to ability to manage labs 15:26:54 not do full stuff - perhaps right in the first place 15:26:55 fdegir: jenkins can tell MAAS to deploy a particualr install in community lab 1/2/3 15:27:08 that I understand 15:27:17 but why do we want to do that? 15:27:30 that's my question 15:27:47 as long as we get CI dedicated resources from lab 15:27:48 BTW - will lab 1 admins be able to run MaaS jobs in Lab 1 only? or how will we split up roles here? 15:27:55 we can do the same with existing CI setup 15:28:11 or once we get resource reserved for CI temporarily by MAAS 15:28:16 we can do the same again 15:28:23 fdegir: as per my undrstanding there is no standard way to deploy Arno release right now in community labs through linux foundation lab and no flexibility or reuse of the comunity labs 15:28:34 iben - we already have the second point - just fyi with the auto deployer on FUel/Foreman 15:28:51 hmm.. i dont know about that narinder 15:28:56 narindergupta: and that is where I think maas can help us 15:29:04 Can we organise a small team to drive the PoC and figure the usage model questions? We need to get to other topics. 15:29:05 no flexibility or reuse of the cocmmunity labs 15:29:06 i can’t use fuel/forman today to push an os to my lab 15:29:12 #info iMSWynne (Michael Wynne) will help drive the PoC from Intel side. 15:29:12 i don’t use physical 15:29:14 that is our first problem to solve 15:29:21 at least for me personally 15:29:28 we don't have visibility of the labs 15:29:28 Iben - what do you mean "push" your lab - are you talking about nightly automated builds 15:29:29 or usability 15:29:37 or doing a deployment through jenkines 15:29:42 deploying stuff on them via MAAS is secondary 15:29:45 for me again 15:29:46 since a deployment and test in the CI pipelines works fine 15:30:01 ok - well - i would say 15:30:03 nightly would be great - how about manual automated builds? say i want to run arno in a vm? 15:30:04 and I'm not against any next steps 15:30:12 just trying to get our priorities aligned 15:30:12 that if the goal is to provide a view to the lab usage and what is deployed int he communicty 15:30:20 we can get away with that without using a full metal deployer 15:30:32 again Iben 15:30:35 FUEL already has this 15:30:40 or I want to give someone else permission to use some capacity in my lab? 15:30:43 you can deploy to metal or a libvirt/KVM automatically 15:30:52 Guys, I think the POC will help answer a lot of these questions. Let's see how it goes and get answers from the activity. 15:30:53 well -= that capacity question is a Metal issue in your lab 15:31:09 i would not be in a position to put a regional controller in LF that would allow Resource allocation in a community lab 15:31:11 trevor_intel: I think it's a good idea to have POC team 15:31:13 we need to be able to do this centrally and with shared visibility 15:31:22 i think the "Use cases" need to be cleaned up cause there are "alot of them" and they are all good 15:31:26 with clear priorities 15:31:38 but as Fatih said - the "purpose of the Proof of concept" remember its a proof 15:31:44 then we can evaluate different cases 15:31:47 cause so far i havent seen in the links anything that we cant do today 15:31:47 and see the value 15:31:57 so im trying to see the "new concept" we are trying to prove 15:32:04 since we know the reginal MAAs thing will work 15:32:14 we know we can auto deploy from Jenkins with what we have today 15:32:21 we have support for BM and nested environments 15:32:40 so i just trying to see how if we combine this it comes as a "new thing" that we can sing about :) 15:32:51 i think its a good idea to have a session to nail down what we want to see / accomplish 15:32:54 Lets stop here please 15:33:05 Who will help drive the PoC? 15:33:22 iMSWynne from Intel side 15:33:49 I definitely want to signup as CI/automation rep 15:33:51 the spirent lab is available - we can setup the cluster controller there with some dedicated networks 15:34:06 I think we need somebody to lead it 15:34:23 someone with MaaS experience? 15:34:23 Volunteer? 15:34:36 who know the desired use cases best 15:34:43 since it woulud seem it would be theirs to drive 15:34:45 I volunteer narinder !!! ;) 15:34:49 haha 15:34:51 trevor_intel: iben: i know MAAS 15:35:01 Ok Narinder? 15:35:04 +1 narindergupta 15:35:13 +1 15:35:16 +1 15:35:25 +1 15:35:46 +1 15:35:56 I'd like to lurk but will not be useful. 15:35:59 So Narinder will be in charge or clarifying the use cases and outlining the "proof point" 15:36:01 +1 15:36:03 In general 15:36:33 narindergupta: do you accept the nomination? 15:36:46 trevor_intel: yeah sure i will work towards it 15:37:13 #info narindergupta will lead MAAS PoC 15:38:02 narindergupta: lets get an update next week? 15:38:17 #topic Rls B planning for Pharos 15:38:22 trevor_intel: sure 15:38:30 narindergupta: can you please start an etherpad or somethin 15:38:33 for you MAAS experts - here is my first qestion - having tried it - since MAAS expects to have a) control on the BM via PXE and b) expects pre-defined images for deployment - how do you get around having the FOREMAN and FUEL images being booted/intalled with config and then they will bring up their own PXE server to boot the remainder.. I wasnt able to find how to tell MAAS to stop PXE and let HW control go down to the other installers 15:38:33 without rebooting the nodes a couple of times (which breaks automation). 15:38:49 MAAS topic has ended 15:38:54 sorry - was typing 15:38:55 next time . 15:38:56 :) 15:38:59 :) 15:40:07 We have a few weeks to plan requirements/milestones/dependenceies for Pharos to get to rls misetone "C" 15:40:57 we have collected very basic/initial reqs for CI parts 15:41:15 and added that to Pharos Release B spec 15:41:17 Should we list out all teh topics to start? 15:41:45 trevor_intel: I think we had a list from before the summer 15:41:56 Pharos B deliverables 15:42:08 compliance, lab spec, etc. 15:42:09 So lets prioritise 15:43:12 CI usage of community labs is one where there has been some requirements documented 15:44:06 What other lab requirements do we need from other projects? 15:44:26 For CI part of the specs, how is it related to releng project? 15:44:51 Sorry I am new to the projects 15:45:11 wshao_: I don't see any mention to any project 15:45:15 in those reqs 15:45:29 it is just plain old CI 15:45:35 Ok 15:45:52 #link https://wiki.opnfv.org/pharos_rls_b_plan 15:46:06 meaning that any activity that is running in automated fashion requires certain things 15:46:14 and that's what we tried to capture 15:47:38 fdegir: lets go through them? 15:47:53 shall provide means/utilities to (re)configure them automatically when necessary 15:48:21 Is this not what MAAS gets at? 15:48:23 automatic (re)configuration: this is actually valid for automated activities and for developer 15:48:43 trevor_intel: that was my other question regarding MAAS which I've forgotten to ask 15:48:50 can MAAS do that? 15:50:16 some more info about this 15:50:24 Installers should handle that. We should not require MAAS. It is just one of the ways to do Bare metal provisioning 15:50:26 For now at least in my mind I have MAAS against that requirement ... can that be a goal of the PoC to prove out? 15:50:28 Peter Bandzi written a utility to do this for LF lab 15:50:44 wshao: it is not installer thingy 15:50:58 it is done via UCS if I'm not terribly mistaken 15:51:03 for LF lab 15:51:08 https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=tree;f=utils/lab-reconfiguration 15:51:23 Community labs are not UCS 15:51:36 yes 15:51:46 the reason I mention what we have is to give an example regarding this requirement 15:51:54 and pehaps if LF infra grows it will not be either 15:52:04 each lab should provide mechanisms to do what we are doing for LF 15:52:12 UCS or not 15:52:22 question if i can? 15:52:24 fdegir: why not? Are we talking about re-configure a stack? 15:53:03 lab infra is something handled by pharos 15:53:04 with the exception of doing a Virtual Setup - wher eyou can specific / fake out the networking, how do you plan to deal with the physical environments in the labs 15:53:05 not via installers 15:53:20 those that use VLANs, dont, specific switch settings? since each installer will need different configurations. 15:53:36 if you are going to switch between installers doing deployments (from MAAS or not). 15:54:10 will you require that community labs provide a generic (one configuratnoi that meets all installers) approach for hardware networking cofiiguration 15:54:19 or will we just mandate all virtual deployments or? 15:54:44 my point is that while you can use whatever PXE/TFTP boot server combo with IPMI you want to the hardware and boot OS 15:54:45 All community labs need to adhere to teh Pharos spec 15:54:59 that is easy - but eac installer bring different topologies so how to deal with that change? 15:55:18 thx trevor- and the pharos spec is matches the changing reqs of the installers coming online? 15:55:59 or just the reference arch? 15:57:01 Installers should beable to be confgured to a baseline specificaiton of compute, netowrk and storage 15:57:27 hmm. ok - well ni the LF / ARNO release, we have two toplogies 15:57:36 and this required a script and networking setip to support both 15:57:45 if we are doing to say "they should just support it" fine. 15:57:57 i bring it up that community labs will not have the resources necessarily to do this 15:57:59 thats all 15:58:24 That is what we have to define ... the resources needed 15:58:39 And put this on the labs to support 15:58:54 hmm. ok.. seems chicken and egg but lets go with it 15:58:54 :) 15:58:57 That IS the Pharos spec I think 15:59:03 just to highlight again; ability to configure or rollback labs to initial/original state stuff is valid for non-CI uses 15:59:40 and I'm not sure if we should expect everyone to be able to do this if labs are not going to provide it 15:59:52 or provide a dedicated person to do this for each and every request 16:00:01 that's why the scripted configuration is there 16:01:00 Next requirement: labs shall be integrated to (yet to be defined/created) Pharos Booking System so they be booked by CI or individual developers 16:01:15 Any ideas on this? 16:01:57 Or this one: labs shall be integrated to (yet to be defined/created) Pharos Dashboard so the availability of them can be tracked realtime 16:02:23 hmm.. well 16:02:35 for this part i have been workng on a lab dashboard for our own management 16:02:47 this also depends on maas poc perhaps 16:02:51 There are other less difficult requirements ... the 3 mentioned are the hard ones ... agree? 16:02:52 but nothing that could be put forth. 16:03:07 agreed- cause the scope is a bit interdepedent. 16:03:25 yes 16:03:31 for example - the last one could mean that we just want the monitoring of the lab usage on a HW level, or more granular 16:03:40 and the permission and admin/operator cases come into play 16:03:46 so the scope could be really really big or simple 16:03:55 lmcdasm: :) 16:03:59 i think we need to refine those three biggies into something more concrete and manageble 16:04:10 so that we can deliver something specific.. and with expected results / features 16:04:10 Need help to define next steps 16:04:11 agree 16:04:14 agree! 16:04:21 regarding pharos spec can there be only one choice? Or can you, for example, say here are 3 ways: access ports and trunk ports or vxlan 16:04:23 why they need to be integrated to dashboard/booking system 16:04:35 iben - its a possible method 16:04:37 we should offer flexibility 16:04:37 I can try to add some more details 16:04:48 we can say " you need to support a subnet of these tech" or something like that 16:04:55 and have that match the various installer requirements. 16:05:00 (which we would need to get / fetch) 16:05:07 i agree with standards but we can’t say everyone should do things exactly the same way 16:05:32 im a bit fuzzy as to how MAAS (Not the POC but in general) fits inot the OPNFV picture, since it would be an installer on par (not undercloud) to FOREMAN/FUEL at thisp oint i would think - not discounting the POC ideas at all 16:05:33 no but we need a minimum, otherwise teh concept of Pharos goes away 16:05:38 and everythign is proivate 16:05:46 but in terms of getting requiremetns, since MAAS would have to fit the mold for FOREMAN and FUEL set so far 16:05:55 agreed trevor 16:06:05 and we have the ref architecture and some structure alrady 16:06:28 we have a configuration spec required to run CI, i think we should look to expand on what we have with some 'new things" that lead towards a goal (MAAS POC or something else) 16:06:34 rathe than bite off too much to start 16:06:35 I think we all grap the problem :) 16:06:38 grasp 16:06:39 with something 'new" 16:07:06 for example 16:07:18 rather than 'montior,view and deploy" to all community labs from central case 16:07:35 what about for milestone - we have "we have connectivity from a centreal spot" to view HW resources 16:07:36 (stop) 16:07:47 if we get further we can add "and you can order/delte/move/etc" 16:07:55 but getting a view would be enough to start i think 16:08:19 and then we can look at SW / tools to do this (im in favour of a Cloud-OSS myself - openNMS works nice for running my labs) but it can start the discussion 16:08:28 and maybe we can cut down on the big ones like that 16:08:30 ? 16:08:34 * lmcdasm just an idea 16:08:51 lmcdasm: do you want to write up ideas for the milestones on Wiki? 16:08:57 about config spec: we have basic examples for fuel and foreman 16:09:11 i can try to water down what is there 16:09:13 which can perhaps be used as base for the other installers 16:09:27 although the link you sent and the lines you quote in this IRC dont match so i might be looking at the wrong page 16:09:28 and then labs can take those things to see if they can come up with corresponding stuff 16:09:42 we can yes - i think its a question of how we want to position 16:09:48 are we saying "these are the rules and you must" 16:09:54 or "here is things you can and try this" 16:09:55 lmcdasm: CI requirements are here https://wiki.opnfv.org/pharos_rls_b_spec 16:10:01 together with others 16:10:03 thx Fatih for the link 16:11:39 Trevor - i can try it sure - so far i have the paros_rls_b_plan and b_spec doc 16:11:50 I htink we are saying here are some basic rules that you must for purposes A (e.g. CI) B (e.g. dev) C(...) AND here are some things you can try for D E F 16:11:51 are there any other inputs i should look at to come up with targets? 16:12:10 ok - sounds good trevor 16:12:15 lmcdasm: that is is so far 16:12:37 ok... so put and action on me ot have some tangilble targets baed on the spec 16:12:52 and ill create a wiki and come up with a "thing we can see/measure/" for some of the spec 16:13:09 at the very lesat it will start converstaion since i might not "guess" the motive of each of the lines 16:13:13 but it might help! 16:13:47 lmcdasm: will try to clarify those lines, especially CI lines for you 16:13:56 #action lmcdasm to draft draft of milestones based on spec requirements 16:14:00 and make them simpler 16:14:23 that will help us get going for sure 16:14:24 narider_gupta 16:14:43 what is a good time for you - since i would like to pick your brain some more about the "Ragional Controller" ability and functions 16:14:55 do you have availability tomorrow / friday for a 30 minute session ? 16:15:23 fdegir: what else can we do today? 16:15:39 trevor_intel: I think this is a good start 16:15:43 we have 2 names to bug 16:15:51 for POC and reqs 16:16:05 indeed ... and bug we must 16:16:19 the other stuff we had like having labs to provide their capabilities in common format 16:16:28 and for installers to provide their reqs in common format 16:16:31 the template stuff 16:16:56 at least we know that we should have them for B release 16:17:06 we just need to define the templates 16:17:13 and then expected information from labs/installers 16:17:15 Should I try to thow them against some miletones? 16:17:22 requirements should use specifc type of language - right? should, must, could 16:17:26 which we have examples from either arno or from current installers 16:17:36 they shall use shall I think 16:17:56 trevor_intel: can't say if they should be tied to milestons 16:18:01 but they should at least be in the backlog 16:18:10 well.. 16:18:14 my two cents 16:18:17 But that is expected for rls plan? 16:18:38 the plan is high level - we should generate a requiremnts document with the milesontes 16:18:45 (fatih and i will take a first stab and send that around) 16:18:56 iben - you are right - if we do "requirements" then should,must, etc will be used 16:18:57 Ok perfect, thanks! 16:19:02