16:01:33 #startmeeting OPNFV Pharos 16:01:33 Meeting started Wed Dec 2 16:01:33 2015 UTC. The chair is trevor_intel. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:01:33 Useful Commands: #action #agreed #help #info #idea #link #topic. 16:01:33 The meeting name has been set to 'opnfv_pharos' 16:01:52 bertrand_orange1: welcome 16:02:05 * fdegir_ is having trouble with irccloud 16:02:25 trevor_intel: yes 16:02:28 writing the mail 16:03:26 fdegir: ok I will talk to jack today, he may come on this meeting 16:03:27 #info Fatih Degirmenci 16:03:35 #info Narinder Gupta 16:03:40 #info trevor_intel 16:03:50 #info Weidong Shao 16:04:07 # Bertrand Le Lamer 16:04:32 Per yesterday release meeting we need to focus on support for CI/release 16:04:36 #info Bertrand Le Lamer 16:04:44 agree? 16:04:58 What other topics? 16:05:25 nothing that is more pressing than B-release resources 16:05:28 for me 16:05:45 if the time allows, we can chat about MAA Pilot 16:05:47 MAAS 16:05:50 and 16:05:51 LAAS 16:06:00 Ok lets start ... 16:06:07 and before that, perhaps POD config stuff 16:06:10 forgotten that 16:06:29 #topic Lab support for CI/releaseB 16:06:41 oops 16:07:07 fdegir: POD config stuff first? 16:07:17 yep 16:07:20 ok 16:07:38 #topic POD config opens 16:08:08 fdegir: Status of LF PODs? 16:08:56 trevor_intel: I meant yaml based POD config files 16:09:00 but anyway 16:09:07 here is LF POD status 16:09:30 Sorry I'm late. I had to use the web IRC client 16:09:46 ashyoung: NP 16:09:47 #info LF POD2 is operational 16:10:05 #info pbandzi is working on reconfiguring LF POD 1 back 16:10:54 #info 4 new LF blades will be taken into production 16:11:12 I think that's all for LF HW 16:11:32 ashyoung: I'm having trouble with irccloud as well 16:11:53 #info I am working on settign up training for support of UCS 16:12:01 +1 16:12:27 trevor_intel: have you talked to pbandzi? 16:12:43 he has this simulator which might be useful for playing with it 16:13:21 fdegir: I haven't yet ... I will but I think we need something more formal too 16:13:43 that's too 16:13:52 can we move to next topic? 16:14:12 yes 16:14:24 config files? 16:14:32 yes 16:15:06 Jonas made a proposal 16:15:53 bertrand_orange1: Do you have any update on config file templates or thoughts about this? 16:17:16 #link https://gerrit.opnfv.org/gerrit/#/c/3583/ 16:17:16 fdegir: This is also somethign we will disuss on Thursday as I undertand 16:17:46 I think it would be good if pharos can comment on it since there are many things to discuss tomorrow 16:17:54 not sure if we'll have enough time 16:18:48 fdegir: help us here ... ok what can we agree or decide on now? 16:19:11 did you have chance to look at it to the proposal? 16:19:53 when I said we should comment, I meant comment on the patch directly 16:19:55 fdegir: yes but not sure next step 16:20:08 if you're happy with it, you can +1 to the patch 16:20:38 but I think people who are workign hands on with the labs/installs need to comment 16:20:57 I agree with that 16:21:07 but at least we can say this is the way to go 16:21:14 the details must be sorted out 16:21:24 if that patch sits there, it will not move on 16:21:30 So who else on this call agrees? 16:21:39 It can't just be up to me 16:22:01 you represent the majority 16:22:23 no other pharos committer has been in this meeting if I'm not terribly mistaken 16:22:51 since that file will end up in pharos repo 16:23:24 perhaps we can pull morgan_orange to here 16:23:28 morgan_orange: are you there 16:23:43 fdegir not far 16:23:43 So what I can do is prod the Pharos committers 16:23:59 prod? 16:24:18 push 16:24:41 trevor_intel: sorry Trevor no update in lannion at the moment 16:24:43 that would be good 16:24:59 morgan_orange: I've seen you commented on config patch sent by Jonas 16:25:00 I am not comfortable being the only one to make important decisions 16:25:17 are you ok with the pod config files which falls into pharos area 16:25:49 morgan_orange: ^^ 16:26:03 you mean pharos has to work on the pod_config.yaml? 16:26:17 morgan_orange: pod config files will end up in pharos repo 16:26:26 yes 16:26:28 makes sense 16:26:31 morgan_orange: based on example provided by Jonas 16:26:39 so if we get an agreement regarding that 16:26:46 I think it was somehow what was imahined in the Jira asking for a json description... 16:26:51 we can request the POD owners to create those files for their labs/pods 16:26:57 that is pending for a while 16:27:01 jira says json/yaml 16:27:22 but yes, it shall be up to each lab owner to provide such file 16:27:27 yes 16:27:30 the difficulty is to agree on what we put into 16:27:39 if we consider MAAS export there are lots of lines... 16:27:50 the thing is, the jira stuff has been there for about 2 months 16:27:59 and proposal from Jonas is the only thing we have 16:28:40 https://jira.opnfv.org/browse/PHAROS-30 16:28:58 fdegir: perhaps we set a deadline for comments or alternative proposals? 16:29:11 so Jonas kind of fixed that story partly 16:29:19 * DanSmithEricsson seems n. america IRC Is back online 16:29:28 * DanSmithEricsson wave 16:29:37 and this one 16:29:38 https://jira.opnfv.org/browse/PHAROS-34 16:31:14 Pharos-30 is POD config 16:31:24 Pharos-34 is deploy config 16:31:36 yes 16:31:43 that's what Jonas proposes 16:31:49 templates 16:32:04 There will be buil config and test config too ... but I don't think those belong to Pharos? 16:32:12 build 16:32:14 agree? 16:32:18 just the pod description 16:32:32 if you look at Jonas patch https://gerrit.opnfv.org/gerrit/#/c/3583/ 16:32:32 PHAROS-31 and PHAROS-35 are for the real files based on templates provided by PHAROS-30 and PHAROS-34 16:32:44 only pod_config.yaml should be managed by lab owner 16:32:53 morgan_orange: that's right 16:32:55 as Jonas cannot guess all the lab configs 16:33:04 yep 16:33:14 so it is important for installers and lab owners to look at the patch 16:33:41 fatih - a question though 16:33:47 DanSmithEricsson: yes 16:33:49 the other config files are installer dependent, first one is generic to all, 2 and 3 are installer specific (that is why it is in a folder) 16:33:50 ok so I can hasstl teh lab owners about this and give a deadline ... is that good enough? 16:34:02 about this patch.. am i to understand that this config will still be held in a pharos or fuel repo somewhere? 16:34:14 since i cant really put the IPMI/ILOM IP and MACs in a public place 16:34:56 trevor_intel: good. what is the deadline you're thinking? 16:34:56 DanSmithEricsson: pod config will be in pharos repo 16:34:56 DanSmithEricsson: deployment config could end up in releng 16:34:56 DanSmithEricsson: or genesis 16:35:03 DanSmithEricsson: that is not decided yet but Jonas was mentioning releng since the CI is driven from releng repo 16:35:15 DanSmithEricsson: so we can pass the deployment config to CI from releng easily 16:35:23 fair enough - thanks.. i think and important consideration would be that from a standpoing of the lab 16:35:24 DanSmithEricsson: but I don't really care tbh 16:35:28 fdegir: 1 week 16:35:33 the configs can only go so far as the Jumphost in terms of public info. 16:35:53 trevor_intel: ok, then we discuss during next pharos meeting 16:36:12 the format and structure sure, but we cant have a central place where all the lab info is avaiable,, since if someone cloned the repo they only need one key and we are open - sorry - we got attacked yesterday so its prominent on my mind. 16:36:17 fdegir: ok 16:36:18 great! 16:36:36 DanSmithEricsson: would be good to see this comment on the patch as well :) 16:36:43 will add it 16:36:44 DanSmithEricsson: hmmm, ok 16:37:17 that kind of sensitive info should be excluded from config 16:37:21 the challenge is that while the PODs are isolated off, theere is still an uplink to a switch thus surrounding infra is vulnerable to someone who gets in 16:37:23 yup 16:37:35 and now, since we were hit, i will have to enforce this 16:37:43 DanSmithEricsson: I got you now 16:37:44 and clear out our lab infra ip info off OPNFV now 16:37:46 :) 16:37:48 DanSmithEricsson: you're a lab owner!!! 16:37:56 DanSmithEricsson: so you have to comment on it 16:37:59 cause if someone hit our lab, then they can ge to the switch, they "could" get further 16:38:00 yu 16:38:08 yup.. will do.. no more comment needed today 16:38:10 sorry .. 16:38:27 DanSmithEricsson: sorry for what! happy to have you here 16:38:41 overriding the flow of the meeting with side comment 16:38:43 and your comments are always welcome 16:38:43 :) 16:38:44 thx 16:38:55 +1 16:39:02 trevor_intel: I think we covered as much as we can for config stuff 16:39:10 Can we discuss release support now? 16:39:14 + 16:39:19 1 16:39:20 1 16:39:54 +1 16:39:56 #topic CI/release lab support 16:40:21 the idea with release support to scale our infra so we can have capability to test different combinations 16:40:34 apart from supporting this 16:40:48 ease the work done by test projects by protecting them from non-working deployments 16:41:07 This is current allocation of labs https://wiki.opnfv.org/pharos_rls_b_labs 16:41:12 that results in separating installer CI stuff from platform/test/release CI 16:41:23 and just this one doubles the needed hw 16:41:35 if we forget about all the combinations 16:42:09 we have 4 installers and we need 4x2 PODs; 1 for installer CI and 1 for platform CI, per installer 16:42:23 and if you add combinations, then the number grows 16:42:36 obviously we will try our best to utilize what we have as much as possible 16:42:51 so that's why I put the number 8 out there 16:42:55 8 PODs 16:43:33 and this is the proposal for Brahmaputra CI process: https://wiki.opnfv.org/octopus/brahmaputra_ci_process 16:43:39 it doesn't go into details 16:44:02 fdegir: can yuo clarify again the meaning of installer CI and platform CI? 16:44:07 ok 16:44:18 currently we run stuff like this 16:44:31 we first build installer using latest in installer git repo 16:44:41 and then we try to deploy using the latest built ISO 16:44:52 and if it fails, it stops test execution 16:45:10 the problem here is that test projects depend on installers in this setup 16:45:23 and since we always try to deploy using an unknown version of the installer 16:45:28 we fail time to time 16:45:43 the first picture on the wiki: https://wiki.opnfv.org/octopus/brahmaputra_ci_process 16:46:05 what is proposed there is not to use the latest on the pods used for testing 16:46:13 but use a known/working version of the installer 16:46:23 so test projects can have results next day when they come into work 16:46:24 Platform CI is what test projects use for release testing? 16:46:35 trevor_intel: yes 16:46:59 the installer CI will give feedback to installer projects 16:47:09 currently both are mixed 16:47:14 Installer CI is what is used to build and deploy the installers 16:47:14 used by installer and test projects 16:47:21 trevor_intel: that's right 16:47:33 platform CI will start from deployment and do the rest 16:47:42 if there is no recent version of the installer 16:48:03 we can perhaps skip the deployment totally (this depends on test projects not messing up with the deployment) 16:48:17 this shortens the time 16:48:27 reduces the risk of having non-working deployment 16:48:36 and opens up space for us to do feature testing perhaps 16:48:48 if you think a failure in deployment 16:49:00 that costs at least 8 hours 16:49:17 4 lost hours during the night and 4 lost hours for redploying the pod for testing 16:49:19 Question: all the PODs beingused by testing (yardstick/functest) for development ... why can they not be repurposed as production CI? 16:50:17 well 16:50:22 thats an interesting question 16:50:25 since for Yardstick for example 16:50:31 i think (have to chekc with Ana for sure) 16:50:44 Yarstick is using at least 3 PODs today 16:50:45 but they want to have Yardstick measuring everything else as it comes up and goes down 16:50:46 ya 16:51:10 Functest uses only LF POD2 so far... 16:51:34 apart from using them for real development work 16:51:35 we just started working to use Orange POD 2 (beta joid) 16:51:42 some PODs might not conform Pharos specs 16:51:53 some PODs might be incapable of fetching artifacts 16:52:07 due to low bandwidth, access problems, etc 16:53:21 fdegir: makes sense ... but isn't it up to Yardstick to say they don't meet the requirements? 16:53:26 btw, I'm not against this idea 16:53:44 partly yardstick 16:53:50 but if they can't access google 16:53:55 there is nothing yardstick can do 16:54:06 that's why they deployed arno sr1 in some labs in China 16:54:15 and they don't redeploy 16:54:20 I get the impression our current resources are not being used as effectively as possible ... just adding more may not solve the problems? 16:54:38 actually we are not asking for adding more 16:54:43 we're trying to match the numbers 16:54:52 morgan_orange already have the table 16:55:06 and we need to fill that table using already available pods 16:55:25 https://wiki.opnfv.org/brahmaputra_testing_page 16:55:40 fdegir: means the test coverage/dependency matrix? 16:55:48 the first table on the page 16:55:52 listing pods 16:56:13 sorry 16:56:26 but there is a requirement to all installers that they support internetless installations 16:56:38 so the access to google, shouldnt come into play 16:56:43 DanSmithEricsson: I think it changed 16:56:47 lab should be able to implement either option 16:56:52 DanSmithEricsson: scripted installations are allowed 16:57:02 DanSmithEricsson: apart from that, we store installer ISOs on google 16:57:09 DanSmithEricsson: so even if all the installers have ISO 16:57:17 are allowed - but the requirement from SR1 is that a lab must support a private installation 16:57:22 DanSmithEricsson: we need an artifact repo for China 16:57:31 they threw this away am i to understand? 16:57:32 DanSmithEricsson: or mirror in China 16:57:52 DanSmithEricsson: there was a TSC decision to allow scripted installs if I'm not mistaken 16:58:10 DanSmithEricsson: or at least a discussion with no major objection 17:00:53 fdegir: back to mrgans table ... so there are x4 PODs for each stable version of deployers ... is this what you would consider as platform CI? 17:01:06 trevor_intel: yes 17:01:08 Install methods available are part of CI configuration right? Is there a full matrix yet of pod configurations needed? 17:01:12 means today there are x4 platform CI PODs 17:01:27 planned... 17:01:39 yep, that table shows the needs 17:01:40 only LF POD2 is performing automatic testing 17:01:46 So if you number of x8 is true ... we only need to allocate anohter x4 PODs to Platform CI? 17:02:06 that's right if morgan_orange doesn't object 17:02:18 and that's to start with 17:02:36 if we see they're not utilized, we can surrender the ones we don't use 17:02:48 the difficult is "only need to allocate" 17:03:02 Ash offered up 5 pods yesterday too. Is anyone working yet to see if these meet config needs? 17:03:18 Well, my definition of a POD was off :) 17:03:30 It's one POD 17:03:38 :) 17:03:48 I put in a request to supply more 17:04:02 if we speculate now 17:04:22 apex uses intel pod2 and we're getting lf pod1 rebuilt 17:04:27 IF we can get 1 POD from Dell, Huawei (SC), Intel .... that is 3 17:04:38 fuel uses lf pod2 and DanSmithEricsson will hopefully give another POD for fuel 17:04:50 morgan_orange: you said joid will end up on orange pod 17:05:01 and joid is on intel pod5 as well 17:05:17 orange pos is not bare-metal? 17:05:19 compass is on Huawei SC 17:05:21 target is Intel POD 5 17:05:36 for CI 17:05:41 morgan_orange: then we lack 2 PODs 17:05:49 1 for compass latest and 1 for joid latest 17:06:06 apex and fuel have the pods for latest and stable based on my speculation 17:06:51 BTW, the PODs I'm supplying would be via Supermicro, not Huawei 17:08:10 don't know what supermicro is - as long as they conform pharos spec and we can deploy and test opnfv on it, that's fine 17:08:12 JOID has Intel PODs 5 (production) and POD 6 (dev/test) 17:08:40 understood 17:08:47 trevor_intel: adding this to table 17:08:54 and the apex and fuel ones as well 17:09:55 added 17:10:55 the row latest on the table means that 17:11:12 they will be connected to jenkins and produce stable ISOs every night 17:11:22 and during daytime, they can be used for development 17:11:29 anyone disagrees with this? 17:11:50 when is night and daytime? 17:11:51 I think we should explain that on teh page next to the table 17:12:00 What about compass? 17:12:08 morgan_orange: that's a good question 17:12:15 currently it is based on UTC 17:12:31 trevor_intel: need to fix a POD for compass 17:12:39 i am just waiting on aIP assignment and yes you have your POD 17:12:51 fdegir: Intel POD 8 17:13:14 updating the table with this 17:13:33 We were doing to allocate that to STORPERF but maybe can make another plan 17:13:53 left question marks on all of them 17:14:02 need to make installer teams know about this 17:14:05 FYI POD 3 will be shared by VSPERF, OVSNFV, kvm4nfv 17:14:07 and what this means for them 17:14:30 trevor_intel: I might have 2 of your servers given to another project - but this is another subject 17:14:42 and a big, question once we will have put the stickers on the 8 PODs, who will install them...jenkins jobs are not ready and will require some tuning 17:14:57 In tomorrows meeting we shoudl review all of this and explain ... yes? 17:15:02 well i can help with installs if you need a "grunt" 17:15:04 trevor_intel: yes 17:15:12 morgan_orange: I think for latest case, it is easy 17:15:15 if there is access and you have a plan of what is to be installed where i can help with that 17:15:27 morgan_orange: we already have apex, compass, and fuel running on those pods 17:15:37 morgan_orange: in installer CI mode 17:15:45 morgan_orange: I'll create new set of jobs 17:15:51 morgan_orange: and send for review 17:16:02 ok and David is working with cnaonical boyes on joid 17:16:16 the new jobs will be named opnfv-fuel-daily, opnfv-compass-daily 17:16:18 etc 17:16:24 so goal to have 8 PODs up&running connected to CI before xmas break...sounds ...reasonable... 17:16:33 and your test jobs will be hooked under these new opnfv-* jobs 17:17:09 morgan_orange: jsut to clarify since DanSmithEricsson asks same thing 17:17:19 morgan_orange: what you mean with install? 17:17:25 morgan_orange: they will be installed by jenkins jobs 17:17:44 morgan_orange: whenever we have new stable version of the installer 17:17:53 ok 17:17:56 morgan_orange: if we want to deploy a different config, then this should be done using jobs 17:18:05 we need to create the jobs based on what we know 17:18:16 100% OK 17:18:36 What's the gap on PODs? It sounds like you have it covered. 17:18:37 to me the first task is cloning existing jobs under the names opnfv-* so things are clearly separated 17:18:41 and then work from there 17:19:03 ashyoung: looks like it 17:19:23 but we need plan B as well :) 17:20:42 Just let me know. I will have 1 by end of the day. And I can probably have a couple more early next week. This is all latest HW. 17:21:02 All using latest Intel SSDs too. 17:21:07 ashyoung: will do, thanks 17:21:13 Sure 17:21:18 * fdegir_ has some experiences with latest hw :( 17:21:31 lf pods were kind of tricky... 17:21:41 Also, 10Gb to each VM, if that makes any difference. 17:21:49 Oh 17:22:15 but anyway, we'll see what happens 17:22:16 I'm working with Rebar folks today to put all of the bare metal provisioning in place 17:22:20 Sounds good 17:22:30 trevor_intel: ashyoung: morgan_orange: DanSmithEricsson: debrascott: are we happy then? 17:22:55 I'm happy :) 17:22:57 yes 17:23:00 me too :) 17:23:49 me too :) 17:24:08 Ok to end meeting? 17:24:15 Then pick up tomorrow 17:24:37 #endmeeting