16:01:20 <frankbrockners> #startmeeting OPNFV BGS daily release readiness synch
16:01:20 <collabot> Meeting started Fri May  8 16:01:20 2015 UTC.  The chair is frankbrockners. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:20 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:01:20 <collabot> The meeting name has been set to 'opnfv_bgs_daily_release_readiness_synch'
16:01:25 <frankbrockners> #info Frank Brockners
16:01:41 <radez> #info Dan Radez
16:01:43 <ChrisPriceAB> #info Chris Price
16:01:46 <fdegir> #info Fatih Degirmenci
16:02:12 <ulik> #info Uli Kleber
16:02:43 <HKirksey> #info Heather Kirksey
16:03:44 <trozet> #info Tim Rozet
16:04:26 <frankbrockners> Do we have anyone from FuncTest? Morgan mentioned that Jose might be able to make it...
16:05:11 <frankbrockners> jose_lausuch can you join us?
16:06:08 <frankbrockners> let's get started...
16:06:38 <frankbrockners> #info status updates on functional testing and deployment tools
16:06:55 <frankbrockners> who'd like to go first?
16:07:25 <HKirksey> Bueller? Bueller?
16:07:34 <trozet> I can go
16:07:44 <frankbrockners> thanks Tim
16:08:38 <trozet> #info update on deploy.sh: Fixed the nova-compute issue with virtualization.  We now have full virtualization working.  Not a requirement for OPNFV, but nice to have.
16:09:16 <trozet> #info Submitted the installation guide to gerrit: https://gerrit.opnfv.org/gerrit/#/c/490/
16:09:23 <trozet> please review it if you have time
16:09:38 * frankbrockners started doing so...
16:09:52 * frankbrockners but not yet done
16:09:57 <trozet> #info will focus next on pulling an "Arno" branch on our tools to make linux packages static versions for the release
16:10:26 <trozet> #info deploy and ISO released for Arno will use these "Arno" branches with static package versions
16:10:40 <trozet> thats it from me
16:10:42 <radez> #info foreman iso has been updated that it will do installation with internet connectivity.
16:11:07 <radez> #info I'll update the install guide next as additions to trozet's patch
16:11:24 <radez> #info created a JIRA-45 to track diconnecting the iso from the internet
16:11:30 <radez> that will take time
16:11:41 <radez> that's all from me for now
16:11:43 <fdegir> trozet: I suppose  this means you will point to specific tag/sha1 or something similar to this to refer versions of tools
16:11:48 <jose_lausuch> #info JoseLausuch
16:11:49 <fdegir> not just latest on "Arno branch"
16:11:51 <jose_lausuch> sorry for the late
16:12:49 <trozet> fdegir: will point to a specific branch for each tool.  For instance, Khaleesi "Arno" branch will install a specific version of python, etc
16:13:48 <fdegir> ok
16:14:01 <fdegir> where can we see how this is done? some configuration file?
16:14:29 <fdegir> so it is tagged in opnfv repos for release purposes as well
16:15:30 <fdegir> we can take this offline so we don't slow the meeting down
16:15:46 <frankbrockners> fdegir: let's just do this
16:15:52 <trozet> fdegir: it wont be a config file, it will be code changes, we have documented the current package versions we use for jumphost and nodes on the opnfv wiki
16:15:57 <trozet> k
16:16:25 <frankbrockners> there's also been a discussion on what "release" would mean: see a draft at https://wiki.opnfv.org/get_started/release_and_maintenance
16:16:29 <frankbrockners> ok
16:16:32 * ChrisPriceAB yes labels not branches please ;)
16:16:35 <frankbrockners> thanks trozet and radez
16:17:32 <frankbrockners> #info status update on POD1 deployment?
16:17:46 <frankbrockners> ChrisPriceAB - do you have an update?
16:18:05 * ChrisPriceAB pending incoming communications give me a few seconds.
16:18:22 <frankbrockners> #undo
16:18:22 <collabot> Removing item from minutes: <MeetBot.ircmeeting.items.Info object at 0x1d77dd0>
16:18:28 <frankbrockners> then lets switch to functest
16:18:36 <jose_lausuch> ok
16:18:41 <frankbrockners> #info status update on functest:
16:18:44 <frankbrockners> Jose?
16:18:58 <jose_lausuch> #info vPing has some issues and a lot of hard coded dependencies
16:19:10 <jose_lausuch> #info it relies on existing netwrks and images
16:19:28 <jose_lausuch> #info Im planning to have a common configuration file for all the test
16:19:54 <jose_lausuch> #info needed networks and images will be created automatically when deploying functest environment, and deleted afterwards
16:20:18 <jose_lausuch> #info hopefully ready on monday
16:20:47 <jose_lausuch> #info Tempest: getting less errors, but still troubleshooting
16:20:54 <RayNugent> Folks, do we have a defined critical path of issues that absolutly must be fixed in order to ship?
16:21:20 <frankbrockners> RayNugent: Let's get through the updates first please
16:21:40 <RayNugent> It's a simple yes/no question
16:22:08 <jose_lausuch> I dont find any "critical" issue from my side
16:22:36 <ChrisPriceAB> Ray, you can check here https://wiki.opnfv.org/releases/arno/releasereadiness
16:23:18 <frankbrockners> jose_lausuch: Do we have an understanding of the root cause of the failing tests (i.e. are those because of issues with the test, config issues, sequencing issues, component issues)?
16:24:01 <RayNugent> Chris, of the Nos are any blockers to release?
16:24:04 <jose_lausuch> https://etherpad.opnfv.org/p/functiontestrelease1activities
16:24:22 <jose_lausuch> at the end you can find the issues
16:24:46 <jose_lausuch> for example
16:24:47 <jose_lausuch> 3 x Details: {u'message': u'Multiple possible networks found, use a Network ID to be more specific.', u'code': 400}
16:24:56 <jose_lausuch> this is something because of wrong configuration
16:25:00 <jose_lausuch> as an example
16:25:04 <jose_lausuch> they are identified
16:26:19 <frankbrockners> Could we get this into a proper table on a wiki?
16:26:21 <RayNugent> OK, it's great that issues are identified! My question is do we know what absolutely has to be fixed, by who and in what order - a critical path?
16:26:58 <jose_lausuch> in some cases I dont know, and I might need support from someone to fix it
16:27:18 <jose_lausuch> or I can spend time myself to find the root cause and fix it myself
16:27:28 <frankbrockners> I.e. something that lists test fails because: test itself has bugs, config is wrong (needs fixing), sequence/state issues (needs investigation), component issues (documentation/release note)?
16:27:48 <frankbrockners> ok - let's start with the documentation of the issues first
16:28:01 <frankbrockners> if you already know the root cause - let's document it
16:28:17 <frankbrockners> if you don't know - flag this as well. We'll need to all jointly look into this.
16:28:27 <frankbrockners> Key is that we understand why things fail
16:28:28 <ChrisPriceAB> Is this troubleshooting something we could get more eyes on that will help us out quickly?
16:28:47 <jose_lausuch> yes
16:29:04 <ChrisPriceAB> Ok, sweet.  I'll see what we can do to help you out.
16:29:10 <jose_lausuch> I would say that if someone helps out to see what the tests fail, then we can figure it out quicker
16:29:19 <frankbrockners> ChrisPriceAB: this is the idea of having Jose create the table... once we have the table we can create visibity
16:29:26 <jose_lausuch> Im also busy with vPing and the functest scripts
16:29:36 <jose_lausuch> a lot of tasks in parallel :)
16:29:42 <RayNugent> does it stand to reason that the tests are the root cause since that is the lions share of new code?
16:29:55 <radez> jose_lausuch: if you have links with errors your can send or something I'm happy to help troubleshoot openstack related stuff
16:30:09 <jose_lausuch> yep, please have a look at the etherpad
16:30:11 * frankbrockners Jose - this is very much recognized by everyone here... Thanks for all the hard work driving this forward
16:30:12 <radez> just ping me and we can connect
16:30:16 <radez> ack
16:30:17 <jose_lausuch> sure, thanks radez
16:30:36 <jose_lausuch> :)
16:30:50 <jose_lausuch> I was talking about tempest tests
16:30:58 <jose_lausuch> about vPing is something to be worked and fixed
16:31:14 <frankbrockners> thanks jose_lausuch
16:31:16 <frankbrockners> ok - let's move to POD1 now
16:31:22 <jose_lausuch> but tempest cases are predefined and we cannot (shouldnt for now) changed them, just run them or not
16:31:25 <frankbrockners> ChrisPriceAB: ready by now?
16:31:29 <jose_lausuch> yep
16:31:31 <radez> jose_lausuch: oh yes, I'm refering to the tempest stuff, I wouldn't be any help on the vping sutff
16:31:41 <jose_lausuch> sure
16:31:48 <jose_lausuch> just in case ,)
16:31:52 <ChrisPriceAB> #info POD 1 is deployed with the Arno software.
16:32:15 <ChrisPriceAB> #info issues remain with DHCP leakage across the PODs which inhibits the autodeploy development
16:32:44 <ChrisPriceAB> #info work is ongoing on autodeploy, but we need to see the DHCP issue solved before we can really get the test working
16:32:59 * ChrisPriceAB test == autodeploy
16:33:07 <frankbrockners> #info there is a proposal from Konstantin (see email) to dedicate another subnet - to fully isolate the PODs
16:33:32 <ChrisPriceAB> #info yes, that's a good enough workaround for Arno i think.
16:33:50 <frankbrockners> #info please review Konstantin's proposal and +1 if you're ok. We can get this done today
16:34:01 <ChrisPriceAB> we should preferably solve the issue as it will be prevalent in any labs that do not isolate Arno by subnets
16:34:18 <ChrisPriceAB> but I think we can get to Arno with the workaround
16:34:35 <frankbrockners> agreed. Intel has pods isolated already
16:34:36 <ChrisPriceAB> get to Arno == release
16:34:41 <frankbrockners> should be part of pharos docus
16:35:13 <frankbrockners> #info: Proper isolation of PODs needed. Should be included in Arno release docs (for Pharos docs)
16:35:30 <frankbrockners> ChrisPriceAB: Has any testing been done on POD1?
16:35:47 <ChrisPriceAB> Anyone with insight into how that would affect nested virt environments?
16:35:53 <jose_lausuch> functest you mean?
16:36:05 <ChrisPriceAB> jose_lausuch might have to answer that :)
16:36:08 <jose_lausuch> no afaik
16:36:22 <jose_lausuch> Morgan was focusing on pod 2
16:36:27 <ChrisPriceAB> let me check
16:36:35 <jose_lausuch> I got the access to pod1 as well though, I could try to run some tests
16:37:14 <jose_lausuch> sorry I didnt get, is pod1 a fuel or foreman deployment?
16:37:23 <frankbrockners> pod1= fuel
16:37:29 <jose_lausuch> ok
16:37:32 <frankbrockners> pod2=foreman/quickstacl
16:37:40 <jose_lausuch> who is gonna use pod1 this weekend?
16:37:55 <ChrisPriceAB> The guys are in transit and I don't have a clear statement in my inbox on testing.  I assume basic deploy validation has been done but I don't think the functest is running there yet
16:38:12 <ChrisPriceAB> as Jose stated the focus is on POD2 at the moment
16:38:12 <jose_lausuch> i dont think so either
16:38:17 <frankbrockners> jose_lausuch: ask this question on the alias - just a subset of folks here...
16:38:37 <jose_lausuch> ok
16:38:52 <fdegir> if it's been deployed already, we can trigger the functest jobs on pod1 as we did it with Morgan to see what happens
16:39:10 <frankbrockners> another note: if we reconfigure networking per Konstantin's proposal, IP addressing for POD2 would change completely
16:39:10 <jose_lausuch> we could try the tempest
16:39:18 <ChrisPriceAB> jose_lausuch; I suggest reach out to stefan_berg and he can probably give you a time to work there.  I think he intends to finish autodeploy issues once the isolation is solved.
16:39:28 <jose_lausuch> ok
16:39:32 <jose_lausuch> I'll contact him then
16:40:44 <frankbrockners> agreed: let's do things in sequence: (1) reconfigure networking, (2) autodeploy for POD1 (3) test for POD1
16:41:27 <frankbrockners> but note that (1) above will change POD2 - which might impact current testing
16:41:34 * ChrisPriceAB mm probably worth re-deploying POD 2 after the reconfig... should not be an issue but you never know... ;)
16:41:48 <jose_lausuch> that is fine as well
16:41:56 <jose_lausuch> as long as we know when
16:42:02 <ulik> I have to put a stupid question here.
16:42:32 <ulik> Should we really slow down POD2, while this is bringing Arno forward so much?
16:42:47 <frankbrockners> ack - network config means: (a) LF GW config changes (b) UCS/server config needs to change (c) redeploy of POD2
16:43:08 <ulik> What's the risk on POD2?
16:43:46 <trozet> frankbrockners: we just manually change the IP and the gateway on the baremetal jumphost, then redeploy pod2, the script will figure out the IP for us
16:44:07 <ulik> any risk or just 2 hours work?
16:44:14 <trozet> no risk
16:44:17 <frankbrockners> trozet: agreed - no big deal
16:44:49 <ulik> OK.
16:44:54 * ChrisPriceAB risk mitigation rather than adding significant cost.
16:44:55 <frankbrockners> and a useful exercise
16:45:10 <frankbrockners> if we'd not even be able to change IP addresses on the fly then I'd be really worried
16:45:18 <ulik> But let's work that way. Make sure we don't put any risk on the track leading fast to completion.
16:45:33 <frankbrockners> ok
16:45:48 <frankbrockners> ChrisPriceAB: Any additional updates on POD1?
16:45:49 <ulik> Of course. That was my worry all the time. Why is everything so much dependent on LF setup.
16:46:03 <ulik> We have to eliminate that sort of stuff in R2.
16:46:08 <ChrisPriceAB> None that I am able to report in an accurate enough manner to provide value.
16:46:13 <frankbrockners> ok
16:46:31 <frankbrockners> looks like we're done with the updates for today then
16:46:42 <frankbrockners> anything else we need to discuss?
16:46:43 <ulik> So can we give Ray a chance now?
16:48:49 <ulik> sorry, seems to late...
16:49:04 <frankbrockners> ok
16:49:33 <frankbrockners> looks like we're done for today - next checkpoint is on Monday 8am PDT (BGS weekly)
16:49:45 <frankbrockners> thanks everyone - and have a nice weekend
16:49:52 <frankbrockners> #endmeeting