16:00:48 <frankbrockners> #startmeeting OPNFV BGS Arno release readiness - daily check in
16:00:48 <collabot> Meeting started Wed May  6 16:00:48 2015 UTC.  The chair is frankbrockners. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:48 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:00:48 <collabot> The meeting name has been set to 'opnfv_bgs_arno_release_readiness___daily_check_in'
16:00:54 <frankbrockners> #info Frank Brockners
16:01:01 <trozet> #info Tim Rozet
16:01:05 <morgan_orange> #info Morgan Richomme
16:01:05 <pbandzi> #info Peter Bandzi
16:01:36 <frankbrockners> anyone from Fuel around?
16:02:22 * frankbrockners pinging Chris Price
16:02:45 <pbandzi> for Fuel I know that it is testing inside environment with proxy now
16:03:09 <pbandzi> and today Stefan and Szilard obtained LF access so they can start Fuel deployment there
16:03:46 <frankbrockners> ok -- why don't we get started - and Peter can fill in what he knows about the Fuel deployment incase Jonas/Chris cannot make it
16:04:24 <frankbrockners> Tim, Morgan  - could you start with an update from your end?
16:04:49 <trozet> #info I've been working on a patch that will allow full virtualization of all nodes
16:05:15 <trozet> #info It should be ready soon and then those who want to try out the deploy but dont have the hardware can use virtual nodes
16:05:37 <[1]JonasB> #info Jonas Bjurel
16:06:02 <trozet> #info Morgan mentioned to me today he ran into some issue with network creation in the LF setup.  I see some networks are created, but after that he has trouble making more
16:06:18 <trozet> #info looks like an ODL bug to me, but haven't sync'ed up with Morgan yet to debug further
16:06:40 <radez> #info Dan radez
16:07:00 <trozet> thats it from me
16:07:01 <morgan_orange> #info yes we need to synchronize on that, seems that much of the smoke tests failed due to impossibility to create new network properly (and to delete old ones)
16:07:38 <frankbrockners> radez: any updates from your end (ISO build)?
16:08:00 <radez> #info nothing new, just working on rolling resources into the iso
16:08:24 <frankbrockners> #info any thoughts on when this will be complete?
16:08:55 <radez> haven't really tried to quantify it...
16:09:08 <frankbrockners> ok
16:09:36 <frankbrockners> another Q: where are we on documentation for fuel/quickstack?
16:10:52 <frankbrockners> ups - sorry - i meant foreman/quickstack
16:11:05 <frankbrockners> did not intend to already switch topics
16:11:17 <[1]JonasB> #info Documentation for fuel is pending final review
16:11:30 <trozet> #info I'll put together the user/install guide and get it into gerrit either today or tmrw morning
16:11:42 <frankbrockners> thanks trozet
16:12:00 <frankbrockners> [1]JonasB - could you give an update on Fuel?
16:12:06 <frankbrockners> #info updates on Fuel:
16:12:40 <[1]JonasB> #info We're basically done with all devel. for the Fuel track
16:13:31 <frankbrockners> any updates on where things are from a deployment to the LF lab?
16:13:34 <[1]JonasB> #info: LF-lab transition should have started yesturday - lead by Joseph G, but unfortunately I havent got any progress relport from that.
16:13:55 <[1]JonasB> I will try to get in touch with Joseph tonight!
16:14:13 <frankbrockners> thanks - will try to pull Joseph into our daily synch
16:14:57 <frankbrockners> [1]JonasB: anything else?
16:15:24 <frankbrockners> pbandzi: You mentioned that you worked with Joseph and team - any updates from your end?
16:15:53 <pbandzi> not other than Jonas sent
16:16:04 <[1]JonasB> Thats all I can report since yesturday.
16:16:25 <frankbrockners> ok thanks guys
16:16:46 <frankbrockners> morgan_orange: any news from a functest perspective?
16:17:02 <frankbrockners> (other than the ODL issue already mentioned)
16:17:18 <morgan_orange> #info script for test preparation updated today - will not be able to be fully automated bu will be simple enough (and documented)
16:17:43 <morgan_orange> #info test on POD2 as discussed blocking point seems linked to network management
16:18:13 <frankbrockners> #info morgan_orange: can you briefly elaborate on what "script for test prep" means and why it cannot be fully automated?
16:18:15 <morgan_orange> #info not the same issue on Ericsson POD, jose reached ~10 erros on more than 100 tests
16:18:48 <morgan_orange> #info script for test prep means a script that wil install the tools, get the scenario, set the env variables
16:19:17 <morgan_orange> #info still need to source the rc file from openstack and configuration convention (naming) is different from the installer,
16:19:45 <morgan_orange> #info so need to perform manual operation once on jumphost
16:19:57 <morgan_orange> but once done tests can be fully automated
16:20:10 <morgan_orange> #info but once done tests can be fully automated
16:20:14 <frankbrockners> #info Does this mean that the test setup is dependent on the installer used?
16:20:49 <morgan_orange> #info not the test setup = same procedure but the configuration is different
16:21:17 <morgan_orange> #info e.g. no images preloaded on POD2, testVM already there with fuel, not sure all passwords are the same,...
16:21:37 <frankbrockners> #info Just to confirm: Test tools installation would be a one-time effort - and be independent from the installer. Correct?
16:21:46 <morgan_orange> #info yes
16:22:08 <morgan_orange> #info installed once, run as long as installer do not change default passowrds, ..
16:22:10 <frankbrockners> thanks morgan_orange
16:22:26 <morgan_orange> #info first tests performed from CI on LF POD 2
16:22:49 <morgan_orange> #info errors due to networks (vPing, smoke tests and bench) but mechanism in place
16:23:10 <morgan_orange> #link https://build.opnfv.org/ci/view/functest/
16:23:55 <frankbrockners> thanks
16:24:15 <morgan_orange> #info risk on vIMS suite, I tried to contact Martin and Andrew (suggest to perform the test for them if they could provide the procedure) => no feedback so far
16:24:45 <frankbrockners> ok
16:25:30 <morgan_orange> #info next step: test on POD 1, + fix issue on POD 2, finalize CI + docs
16:25:55 <[1]JonasB> Question: Since Forman and Fuel uses differen virtualization layers for the installer nod on the jumpstart Forman: VB, Fuel: KVM) - how are we dealing with the contradicting kernel modules, I havent started thinking on it
16:26:02 <frankbrockners> #info so key issues seem to be: (1) Network config (ODL issue mentioned above); (2) vIMS test setup and support
16:26:46 <frankbrockners> I thought that each installer would "clean up" after a run - i.e. anything you loaded you'll remove again
16:27:25 <[1]JonasB> So we need to install and un-install kvm/VB every run. Is that even practical?
16:27:51 <pbandzi> i think only kernel modules will need to be disbaled
16:28:08 <frankbrockners> trozet: How are you dealing with this today?
16:28:34 <trozet> we have a clean script that removes virtualbox and the kernel modules
16:28:39 <[1]JonasB> Then we would need root priv
16:28:55 <trozet> jenkins-ci has root
16:30:04 <[1]JonasB> But I dont think we want to run build/deploy in root context, should ci pass root cred? Or will CI do the clean-up?
16:30:23 <trozet> https://gerrit.opnfv.org/gerrit/#/c/460/
16:30:37 <trozet> jenkins-ci executes with sudo
16:30:40 <trozet> for our script
16:30:46 <trozet> he has passwordless sudo access
16:31:00 * ChrisPriceAB can we not get CI to run "clean" before it starts a new deploy?
16:31:09 <trozet> thats what we do
16:32:20 <frankbrockners> [1]JonasB: Could you add your cleanup to 460? Then we'd have a single way to do cleanup
16:32:53 <trozet> then move it to common
16:33:01 <frankbrockners> yup
16:33:05 <trozet> when we both deploy on the same POD
16:33:28 <[1]JonasB> But this implies that no one else uses KVM/Libvirt because we will wipe it away!
16:33:49 <trozet> our clean also powers down the nodes
16:34:04 <frankbrockners> yes - this is the assumption: Jumphost is only installer plus test
16:34:24 * ChrisPriceAB maybe not the best approach trozet.
16:34:33 <trozet> ?
16:34:50 <ChrisPriceAB> well if we run clean before deploy, do we want to power down?
16:35:13 <[1]JonasB> Hmm, no way to test/use autodeploy on othe machines which is using KVM for other stuff
16:35:15 <trozet> yes because deploy auto detects network setup
16:36:06 <trozet> so if those IPs are already in use it will use different IPs and its easier to re-use
16:36:22 <trozet> it parses the inventory yaml file for the nodes and hten ipmi shuts them off
16:37:00 <trozet> they have to be rebooted anyway for PXE boot with deploy
16:37:27 <trozet> so doesnt really hurt anything and reduces network conflicts/chatter during deploy
16:37:43 <[1]JonasB> Have no problem with powering the blades off, but messing with kernel modules ?
16:38:33 <lmcdasm> as a suggestion - we should also look to wipe  (fdisk / parted / etc) the disk as well on successive runs..
16:38:49 <lmcdasm> do that you are sure when you do a deploy successively you have a clean disk partition as well
16:40:14 <trozet> JonasB: if you uninstall the hypervisor it should remove the kernel modules then, if it doesnt you can use rmmod to remove the kernel mods
16:40:42 <trozet> we don't run on the same POD anyway, so this probably doesnt matter for Arno
16:41:35 * ChrisPriceAB ah nice, thanks trozet
16:41:42 <lmcdasm> hgmm. seems we are still stuck with the Jumpstart server being baremetal for one and a VM for the other
16:42:03 <lmcdasm> agreed - for Arno not a big deal (tim) but we need to resolve this at some point :)
16:42:07 * ChrisPriceAB agree with Tim on that this is not gating for Arno.
16:42:25 <[1]JonasB> Proposal, lets figure out a way forward for the SR1 release
16:42:36 <frankbrockners> it is not "gating" for Arno - but we have to resolve this soon thereafter
16:42:45 <lmcdasm> sorry shoudl have said (two different VMs)
16:43:00 <frankbrockners> I do expect that we'll only have a single POD for deploy moving forward
16:43:11 <lmcdasm> hmm..  well that should be a tech discussion
16:43:23 <lmcdasm> cause we already know tht this is not going to work with the native ways we are doing this
16:43:29 <frankbrockners> IMHO we should try the "cleanup" approach for now - until we can agree on a single virtualization tech for the JH
16:43:38 <lmcdasm> we have conflicts from the VB and KVM all the way up to networking approach
16:43:47 <lmcdasm> and touches on the bigger issue of installer framework anyway
16:44:06 <lmcdasm> im not sure we can fix it in a SR1  cause we need to establish (IMHO) a framework for all installers coming
16:44:16 <lmcdasm> or we hit this again and again and everyone (including us :) ) does their own thing
16:44:24 <[1]JonasB> So actually, we need to decide on common technology, eithe VB, KVM, Xen, Docker, ESXI, Oracle,... But we cant let everyone pick their favorite
16:44:46 <lmcdasm> or we dont define it at all
16:44:53 <ChrisPriceAB> It's an OPNFV infra question anyway, SR1 would only really include needed changes to the ISO's.
16:44:53 <lmcdasm> and leave that integratio up the end user
16:45:28 <lmcdasm> agreed Chris which is why im not sure how we could change the ISO in this without basically redoing the JS server architecture.
16:45:28 <frankbrockners> Agreed that it is an LF / OPNFV infra question for now.
16:45:34 <lmcdasm> not something i would want to try in a patch :)
16:46:00 <ChrisPriceAB> Target OPNFV-R2 :)
16:46:06 <[1]JonasB> Is that to say someone else will solve it?
16:46:12 <frankbrockners> for now I would recommend that we go with the pragmatic approach of "cleanup"
16:46:37 <ChrisPriceAB> So guys, what's gating us being highly confident of hitting a release date at this point in time?
16:47:12 <[1]JonasB> frankbrockners: Which means autodeploy is useless for anything else than OPNFV CI
16:47:13 <frankbrockners> ChrisPriceAB - see above: ODL networking issues, vIMS
16:47:21 <lmcdasm> well..  if we arent pinning the FUEL stuff on "must be done on LF hardware" since i undestood the gate was a 3rd party regardless of HW so long as we have a test report
16:47:27 <lmcdasm> we are good to go on FUEL side -
16:47:52 <lmcdasm> if we have to pin to LF hardware (not sure why its soo important there since we will accompany release with test report - i.e hw should be agnostic) then could be a while
16:47:54 <frankbrockners> [1]JonasB: Not sure I follow
16:49:23 <[1]JonasB> frankbrockners: I want to be able to ship autodeploy as part of the delivery, that would relief the support tremendously if consumers could do autodeploy and not have to go through fault prune manual steps.
16:49:56 * ChrisPriceAB that could be an SR1 activity Jonas.
16:49:59 <frankbrockners> [1]JonasB: You can use autodeploy on labs other than LF.
16:51:08 <[1]JonasB> frankbrockners: Not if I assume I own the jumpstart and start messing with kernel config/modules
16:51:13 * ChrisPriceAB ok, do we have the vPing working?
16:51:55 <morgan_orange> #info vPing needs network..
16:51:56 <[1]JonasB> frankbrockners: Infact I think it is a bad habit to assume root priv.
16:52:15 <morgan_orange> as soon as we shall be able to create network and get an IP for the VMs it should work
16:53:06 <lmcdasm> <morgan_orange> - question - has that been done on both ? or just on FOREMAN so far ?
16:53:21 <lmcdasm> i.e POD2 in LF? or on the EriLab as well?
16:53:24 <frankbrockners> [1]JonasB: Well - you need to own/control the JH - as you control the remainder of the HW. Agreed that a solution with non-root access would be preferable - but let's be pragmatic of what we can do in the current situation
16:53:31 <morgan_orange> vPing on Foreman only (and previously on Orange and NTT lab)
16:53:37 <trozet> morgan_orange: couldn't we just redeploy so that vPing can be tested? (with a fresh setup)
16:53:56 <morgan_orange> trozet: yes
16:54:06 <lmcdasm> thx morgan - so for Chris question we have another format to test (FUEL) and networking to do for FOREMAN
16:54:25 <trozet> morgan_orange: ok well let me debug on that server for a bit, then I can ask Fatih or aricg to run jenkins deploy again
16:54:56 <morgan_orange> trozet: ok no problem I have to go now, but just give me a status I will continue tomorrow morning
16:55:02 <trozet> lmcdasm: its not networking to do for foreman.  its an ODL bug
16:55:11 <morgan_orange> for infor public holiday in France on friday...
16:55:25 <morgan_orange> May is not an easy month for releasing...
16:55:38 <trozet> man you guys get a lot of holidays in Europe ;)
16:55:39 * frankbrockners let's all move to France
16:55:43 <trozet> LOL
16:55:44 <lmcdasm> perhaps a comment on root/  etc - we have to recognize that nothing we have so far can pass a security analysis / would be allowed close to production in a operator environment / meet operator DC requuirements (i.e touching on root, open repos in both sets, etc, etc).
16:55:57 <lmcdasm> thx Tim - i hit a similar bug this week with ODL and have talked ot the KARAF release guy to work around
16:56:17 <lmcdasm> (agreed - France it is!)
16:56:54 <frankbrockners> ok folks... looks like we're done for today. See you (at least those who are not in France :-) tomorrow
16:56:58 <frankbrockners> thanks!
16:57:11 <frankbrockners> #endmeeting