15:59:41 <trevor_intel> #startmeeting OPNFV Pharos
15:59:41 <collabot`> Meeting started Wed Jan  6 15:59:41 2016 UTC.  The chair is trevor_intel. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:59:41 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:59:41 <collabot`> The meeting name has been set to 'opnfv_pharos'
16:00:02 <trevor_intel> #info Trevor - Intel lab
16:00:09 <fdegir> #info Fatih Degirmenci
16:00:33 <iben_> #info Iben Rodriguez
16:00:47 <iben_> no lab
16:00:48 <trevor_intel> Agenda topics?
16:01:17 <trevor_intel> 1) Labs 2) Documentation
16:01:21 <jmorgan1> #info Jack Morgan - Intel lab
16:01:27 <fdegir> sounds good to me
16:01:44 <trevor_intel> Ok lets get going
16:01:53 <trevor_intel> #topic Lab status
16:01:54 <iben_> labs is pretty broad - are we focused on installer integration with certain testers?
16:02:28 <trevor_intel> iben_: Everything we are doing now is for rls B :)
16:02:59 <fdegir> trevor_intel: should I type in the status of pods one by one?
16:03:00 <trevor_intel> I think we have to only focus on that
16:03:10 <trevor_intel> Fatih: yes please
16:03:25 <iben_> what is our burn down list?  Get 4 installers working with functest in at least one lab - is there a list for this on the wiki?
16:03:37 <fdegir> iben_: can we please stick to hw side of things
16:04:03 <fdegir> #info LF POD1: currently being reconfigured - will be used for apex on master branch
16:04:03 <iben_> sure - that's fine - trying to agree on scope - thanks! are there hardware issues?
16:04:22 <fdegir> #info LF POD2: operational - will be used for fuel on stable/brahmaputra branch
16:04:58 <fdegir> #info Intel POD2: operational - will be used for apex on master branch - might be having network issues - apex will contact to trevor_intel and jmorgan1
16:05:04 <narindergupta> #info Narinder Gupta
16:05:19 <fdegir> #info Intel POD5: operational - will be used for joid stable/brahmaputra
16:05:27 <fdegir> #info Intel POD6: operational - will be used for joid master
16:05:49 <fdegir> #info Intel POD8: not operational - waiting for jumphost which should be available soon - will be used for compass master
16:05:52 <iben_> the extra jump nodes were added for intel pod 5 and 6?
16:06:05 <fdegir> #info Ericsson POD2: operational - will be used for fuel master
16:06:08 <trevor_intel> iben_: yes
16:06:12 <iben_> great!
16:06:27 <fdegir> I missed one POD?
16:06:34 <trevor_intel> Huawei
16:06:50 <fdegir> #info Huawei POD: operational - will be used for compass stable/brahmaputra
16:07:01 <fdegir> #info some notes
16:07:08 <jmorgan1> fdegir: apex only need to contact jmorgan1
16:07:11 <fdegir> #info Intel POD6 will be connected to Jenkins tomorrow
16:07:17 <trevor_intel> I have updated teh Wiki page with latest status
16:07:22 <trevor_intel> #link https://wiki.opnfv.org/pharos_rls_b_labs
16:07:33 <trevor_intel> 2 questions
16:07:39 <fdegir> #info All the PODs are currently running on master branch
16:07:52 <fdegir> #info PODs will switch to stable/brahmaputra branches once the branching is done
16:07:55 <fdegir> trevor_intel: yes
16:08:04 <fdegir> #undo
16:08:22 <trevor_intel> Fatih: Intel POD 2 issue ... can you explain more to give us a ehad start
16:08:25 <fdegir> #info PODs will switch to stable/brahmaputra branches once the branching is done - based on the table
16:08:44 <fdegir> trevor_intel: need to pull radez and trozet here
16:08:47 <fdegir> pinged them
16:08:58 <trevor_intel> jmorgan1: Can you confirm we are on track to get POD 8 operational today?
16:09:07 <jmorgan1> trevor_intel: yes
16:09:17 <trevor_intel> jmorgan1: thanks!
16:09:24 <trozet> hi
16:09:26 <fdegir> and they're joining
16:09:39 <trevor_intel> nice!
16:09:40 <fdegir> trozet: radez: can you please summarize the network issues on intel pod2
16:10:07 <radez> trozet: has the low dow on it, he's hit the issues and can describe them
16:10:25 <trozet> fdegir: There are 2 issues from my experience
16:10:58 <trozet> fdegir: one is the network bandwidth seems to just throttle randomly, and we have to download the RPM when we deploy and it takes forever
16:11:18 <trozet> fdegir: the other issue is network connectivity seems to randomly drop and come back after a few seconds
16:11:38 <trozet> fdegir: i had to try baremetal deployment a few times because the installer host lost connectivity to the baremetal hosts while deploying
16:12:06 <trevor_intel> jmorgan1: Can you comment, any clues what might be causing these issues?
16:12:08 <fdegir> jmorgan1: is this good enough for you to take a look at intel pod2?
16:12:55 <jmorgan1> i'll take a look today, not sure why
16:13:03 <trozet> trevor_intel: well it could be ports flapping on the switches
16:13:16 <trozet> maybe check the switch log and see if the ports have flapped a bunch of times
16:13:20 <jmorgan1> might be switch or cable issue
16:13:22 <trevor_intel> trozet: what tiem zone are you in?
16:13:25 <trozet> EST
16:13:55 <trozet> for the first issue you can curl our RPM to the jumphost and watch the bit rate
16:14:13 <trozet> the second issue if you just assign an IP to one of the baremetal nodes and ping for a while I bet you will see periods of drops
16:14:28 <jmorgan1> ok, thanks
16:14:30 <trozet> sometimes my ssh to the node locks up for a few seconds then comes back as well
16:15:01 <trevor_intel> trozet: what is best way for Jack to contact you ... IRC?
16:15:22 <trozet> trevor_intel: yeah, or email trozet@redhat.com
16:15:36 <trevor_intel> trozet: ok thanks
16:16:10 <trevor_intel> fdegir: LF POD 1 ... how confident are you that it will be operational this week?
16:16:22 <trevor_intel> fdegir: Do we need a backup plan?
16:16:49 <fdegir> trevor_intel: I think it is pretty crucial to have something for Apex this week
16:17:13 <trevor_intel> fdegir: agree, and I don't have visibility how close we are
16:17:34 <fdegir> trevor_intel: it seems Slovakia is having public holiday today
16:17:44 <fdegir> trevor_intel: and we are waiting Peter Bandzi
16:18:01 <fdegir> trevor_intel: the other competent person is trozet
16:18:07 <fdegir> trevor_intel: for LF POD1
16:18:25 <fdegir> trevor_intel: so it depends on trozet's availability
16:18:32 <trevor_intel> trozet: ?
16:18:39 <trozet> fdegir: I just got a bunch of stuff off my plate, so I have time now to look at LF POD 1
16:18:41 <fdegir> trevor_intel: and urgency of getting bm for apex
16:18:50 <trozet> fdegir: going to try to do it now
16:19:02 <fdegir> trozet: good to hear trozet
16:19:35 <fdegir> trozet: according to Peter, LF POD2 should not be impacted network-wise
16:19:52 <trevor_intel> trozet: Can we get a status update at end-of-day?
16:20:07 <fdegir> trozet: please let me know when/if you finish your work so I can run a fuel deploy on LF POD2 to see if things are still normal
16:20:07 <trozet> fdegir: what do you mean? like i cant screw up pod 2 by messing with pod 1?
16:20:20 <trozet> trevor_intel: will do
16:20:26 <fdegir> trozet: that's right - please keep pod2 working :D
16:20:40 <trevor_intel> Fatih: fussy!
16:21:20 <trozet> hahaha
16:21:23 <trevor_intel> Ok so if there is not line-of-sight by tomorrow I think we need an alternative for Apex ... do you agree?
16:21:27 <trozet> ill do my best not to make things worse
16:21:31 <jmorgan1> topic 3) Pharos project in JIRA
16:21:40 <jmorgan1> please add ^^
16:21:43 <fdegir> here is what pbandzi said
16:21:45 <fdegir> 10:19 <fdegir> pbandzi: will there be impacts on pod2 due to pod1 reconfiguration?
16:21:45 <fdegir> 10:20 <pbandzi> fdegir: pod2 will be without impact
16:22:03 <fdegir> trozet: if you think you're close to messing it up, please stop
16:22:21 <trozet> haha
16:22:25 <trozet> yeah
16:22:25 <fdegir> :)
16:22:31 <fdegir> just being open
16:22:51 <trevor_intel> Ok so looks like we wait for trozet status at EOD and go from there ... any other input on this one?
16:22:53 <trozet> thanks for the confidence boost :)
16:23:03 <trevor_intel> Fatih: as usual :)
16:23:09 <fdegir> I know what you're capable - just joking
16:23:45 <trozet> yeah me too :)
16:23:50 <fdegir> and in parallel to lf pod1, jmorgan1 will take a look at  intel pod2
16:23:54 <fdegir> trevor_intel: do we need plan c?
16:24:25 <trevor_intel> Fatih: c? do we have b?
16:24:51 <trevor_intel> Fatih: I was thinking we need b :)
16:25:10 * ChrisPriceAB has fond memories of plan a ...
16:25:11 <fdegir> b is trozet taking care of lf pod1
16:25:48 <trozet> btw we created a channel for apex, opnfv-apex
16:25:59 <trozet> in case you want to join there or discuss apex stuff
16:26:01 <trevor_intel> Fatih: ahhh yes, as apposed to a which is Peter taking care of lf POD 1
16:26:04 <trozet> everyone is invited except ChrisPriceAB
16:26:14 <ChrisPriceAB> :(
16:26:28 <trozet> haha
16:26:40 <fdegir> trevor_intel: yep
16:26:45 <fdegir> trevor_intel: so if we totally fail
16:26:56 <fdegir> trevor_intel: do we have a POD for apex bm?
16:27:14 <fdegir> trevor_intel: since apex is the only one with no pods atm
16:27:25 <fdegir> no well working pods I mean
16:27:48 <trevor_intel> jmorgan1: Please also send out a status update EOD for ... 1) POD 8 jumphost 2) POD 2 issue ... ok?
16:28:06 <jmorgan1> send status update to who?
16:28:27 <trevor_intel> jmorgan1: tech list with pharos tag
16:28:36 <jmorgan1> ok
16:28:40 <trevor_intel> jmorgan1: I can help with it
16:29:30 <trevor_intel> fdegir: yes lets think about an alternative option
16:29:36 <trevor_intel> any ideas?
16:29:59 <trevor_intel> e.g. is there another Intel POD we can repurpose?
16:30:20 <trevor_intel> e.g. POD 7?
16:30:48 <trevor_intel> POD 7 is allocated for Fuel dev/test
16:30:51 <jmorgan1> yes, POD 7 or 9 are possible. They will be ready today with POD 8
16:31:17 <trevor_intel> POD 9 is needed by STORPERF adn bottlenecks
16:31:59 <trevor_intel> fdegir: Do you think Fuel can get by without POD 7 for rls B (they have up until now)
16:32:19 <fdegir> trevor_intel: I have no idea with POD7 - who uses it for fuel development
16:32:46 <trevor_intel> fdegir: not sure, but it was allocated for that
16:33:01 <fdegir> trevor_intel: this is the first time I hear the existence of pod7
16:33:13 <fdegir> trevor_intel: can you perhaps find out who got vpn access for pod7
16:33:15 <jmorgan1> POD7 ia assigned but not Active
16:33:16 <trevor_intel> Fatih: Its on the Wiki :)
16:33:18 <jmorgan1> https://wiki.opnfv.org/get_started/intel_pods
16:33:34 <fdegir> trevor_intel: so we don't mess up someone elses work
16:34:15 <fdegir> trevor_intel: I only look at your table - ci pod assignment
16:34:32 <trevor_intel> Fatih: See table below that one
16:34:51 <trevor_intel> Fatih: "Community lab resources"
16:35:28 <trevor_intel> Ok so lets ping Fuel to make sure ... tentative plan to reassign to Apex BM ... ok?
16:35:47 <fdegir> trevor_intel: I can definitely say I have no plans for pod7
16:36:03 <fdegir> trevor_intel: fuel uses lf pod2 and ericsson pod2 for ci production
16:36:14 <fdegir> trevor_intel: but anyone using it for development - pinging fuel is better
16:36:18 <trevor_intel> Fatih: POD 7 was for dev
16:36:28 <trevor_intel> Fatih: ok will do
16:36:44 <trevor_intel> Move on to documentation?
16:36:48 <fdegir> +1
16:37:05 <trevor_intel> #topic Pharos documentation for rls B
16:37:09 <ChrisPriceAB> Just a question, is there somewhere we can fun a scenario1 deploy and start doing functional testing for lsoapi for instance?
16:37:13 <jmorgan1> trevor_intel: can we consolidate pod assignment to one place, perhaps link "Community lab resources" table to intel_pods pae?
16:37:22 <ChrisPriceAB> Fun->run
16:38:27 <ChrisPriceAB> I refer to the post-deploy configuration features.
16:38:45 * ChrisPriceAB feels like the spanner in the gears....
16:40:09 <trevor_intel> jmorgan1: This is the place ... link from Pharos page ... https://wiki.opnfv.org/pharos_rls_b_labs?&#status_of_production_ci_resources_for_release_b
16:41:03 <trevor_intel> jmorgan1: Lets review offline ... agree  it should be in one place
16:43:20 <trevor_intel> ChrisPriceAB: We can review all the assignments to see
16:44:21 <trevor_intel> ChrisPriceAB: Means not in current scope of testing projects?
16:45:24 <trevor_intel> ChrisPriceAB: For lsoapi I though they do not have their own test cases yet
16:45:55 <trevor_intel> ChrisPriceAB: Can we take it offline ... want to get to documentation
16:46:24 <trevor_intel> Pharos documentation for rls B ...
16:46:31 <ChrisPriceAB> I know the issues, overcoming them requires a Brahmaputra deploy for them to work on.
16:46:42 <ChrisPriceAB> Follow up offline
16:47:25 <trevor_intel> 1) Pharos spec ... needs updating
16:48:22 <trevor_intel> 2) Key facts ... needs updating
16:48:40 <trevor_intel> 3) Templates?
16:48:50 <trevor_intel> 4) Other documentation?
16:49:32 <trevor_intel> ChrisPriceAB: What are you expecting for rls B?
16:50:11 <ChrisPriceAB> Would like a physical node description and target config.
16:51:05 <ChrisPriceAB> Mandatory is the Pharos description.  Wanted is the target config guide.
16:51:56 <ChrisPriceAB> We can add config to the pre-req sections of the installer config-guides if we don't have a standard yet.
16:52:50 <trevor_intel> Config files we discussed ... 1) INventory 2) Network config 3) Deploy scenarios 4) Test
16:53:55 <ChrisPriceAB> Ok, let's review and discuss what's there and how to plug gaps.
16:55:36 <trevor_intel> Proposal by Tim for common inventory file https://gerrit.opnfv.org/gerrit/#/c/4079/
16:55:46 <trevor_intel> Yardstick - POD Yaml sample https://git.opnfv.org/cgit/yardstick/tree/etc/yardstick/nodes/pod.yaml.sample
16:57:21 <trevor_intel> Jonas Framework proposal  https://gerrit.opnfv.org/gerrit/#/c/3995/
16:57:48 <trevor_intel> Need refresh on this, which projects are responsible
16:58:16 <trevor_intel> Perhaps we can discuss this in tomorrows test meeting?
16:58:53 <ChrisPriceAB> I'll have a look at the current docs tomorrow to prepare.
16:59:36 <trevor_intel> ChrisPriceAB: Thanks, great
16:59:46 <jmorgan1> 3) Pharos project in JIRA ?
17:00:01 <trevor_intel> #topic Pharos project in JIRA
17:00:42 <jmorgan1> I spoke with trevor_intel about adding components and versions (opnfv releases) to make better use of this tool
17:01:05 <jmorgan1> we might want to do a JIRA issue/task scrub to clean up the current items
17:01:28 <jmorgan1> i'd like to use this project to track lab issues (like the Intel pod 2 issues)
17:01:59 <jmorgan1> then issues can be documented via the jira task, and eventually go to release notes or HowTo docs, etc
17:02:02 <trevor_intel> Proposal is to use Jira to track lab ops and support (email is completely unmanageable)
17:03:15 <trevor_intel> jmorgan1: I looked at Jira yesterday and think need some advice form a JIRA guru
17:03:40 <jmorgan1> trevor_intel: I can help with that
17:03:56 <trevor_intel> ChrisPriceAB: Who is a good contact to get help with Jira (setting up labeling scheme)?
17:04:19 <ChrisPriceAB> Mbeirel
17:04:20 <trevor_intel> jmorgan1: E.g. there is no way to add components that I acn find
17:04:33 <ChrisPriceAB> Mark is a bit of a guru
17:04:34 <jmorgan1> Aric is the person I spoke with
17:05:11 <jmorgan1> Aric is the LF jira admin, i beleive
17:06:18 <fdegir> I think Mark and Aric can fix this - continue pinging
17:06:42 <trevor_intel> jmorgan1: I think we have a way forward with this
17:06:51 <jmorgan1> trevor_intel: they just did a Jira upgrade in Dec, so the UI is a little different now
17:07:03 <trevor_intel> Other topics before we close today?
17:07:22 <ChrisPriceAB> #cookies
17:08:20 <trevor_intel> Get back to baking, yay
17:08:57 <trevor_intel> Talk tomorrow
17:09:04 <trevor_intel> #endmeeting