14:01:15 <[1]JonasB> #startmeeting fuelOPNFV No3
14:01:15 <collabot> Meeting started Tue Apr 14 14:01:15 2015 UTC.  The chair is [1]JonasB. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:15 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:01:15 <collabot> The meeting name has been set to 'fuelopnfv_no3'
14:01:51 <[1]JonasB> So could we start with the install issue that mskalski worked on
14:02:08 <[1]JonasB> #topic Fuel install issue
14:02:31 <[1]JonasB> So Michal, I saw you had identified the problem'
14:02:44 <mskalski> couple of minutes ago I send a email where described problem
14:03:10 <fdegir> (I'm currently having 3 meetings at the same time so please action me if you want me to do something)
14:03:28 <mskalski> problem is that on the some ISOs in /var/www/nailgun/ubuntu/x86_64/dists/precise/main/binary-amd64/Packages there is a wrong order of packages
14:03:50 <mskalski> and this order looks to me is a random
14:04:14 <mskalski> on all builds
14:04:27 <[1]JonasB> So what I didn't get was if this sequencing problems was isolated to the opnfv .iso or if it is also present in the origin fuel iso which we build from upstream?
14:04:27 <stefan_berg> mskalski: That's in the yaml file or in the Ubuntu Release file?
14:04:40 <mskalski> in /var/www/nailgun/ubuntu/x86_64/dists/precise/main/binary-amd64/Packages
14:05:01 <lmcdasm> hmm. question - i thought we didnt touch that stuff
14:05:13 <stefan_berg> We do, in order to add additional packages.
14:05:40 <lmcdasm> insert at the bottom yes - but the packages (base_passwd) arent things that we supplant are they?
14:05:54 <lmcdasm> im just wondering if we are "addng in" how the order is changed?>
14:06:11 <mskalski> but all builds generate ubuntu repository
14:06:14 <stefan_berg> We run  apt-ftparchive to add all files from the repo.
14:06:22 <stefan_berg> In our install.sh:
14:06:31 <mskalski> even if don't change anything
14:06:32 <[1]JonasB> So again is the sequencing issue present in the orig iso?
14:07:10 <stefan_berg> APT_REL_CONF="$TOP/install/apt-ftparchive-release.conf"
14:07:10 <stefan_berg> APT_DEB_CONF="$TOP/install/apt-ftparchive-deb.conf"
14:07:10 <stefan_berg> apt-ftparchive -c "${APT_REL_CONF}" generate "${APT_DEB_CONF}"
14:07:10 <stefan_berg> # Fuel also needs this index file
14:07:10 <stefan_berg> cat dists/precise/main/binary-amd64/Packages | \
14:07:12 <stefan_berg> awk '/^Package:/{pkg=$2}
14:07:15 <stefan_berg> /^Version:/{print pkg ": \"" $2 "\""}' > ubuntu-versions.yaml
14:07:17 <stefan_berg> cp ubuntu-versions.yaml $DEST
14:07:20 <stefan_berg> apt-ftparchive -c "${APT_REL_CONF}" release dists/precise/ > dists/precise/Release
14:07:34 <mskalski> order in Packages files looks like random and always different between builds
14:07:55 <stefan_berg> So this is run from the install.sh - has always worked in production here and for my builds, so it is quite weird.
14:08:04 <lmcdasm> #agreed
14:08:06 <stefan_berg> I smell platform dependencies.
14:08:18 <mskalski> so problem can be only visible if base-passwd is defined before base-files
14:08:41 <lmcdasm> i dont see how , since apt-ftparchive is getting from a 'cat" that it would ever be in different order (only when the upstream lists would have changed) or we do something "new"
14:08:57 <lmcdasm> as well, i have not seen this problem with two ISO and three different environments.. im sorry
14:09:13 <lmcdasm> but it works on two servers (SUN and HP) - not counting Nested
14:09:19 <lmcdasm> so its hard for me to catch
14:09:42 <stefan_berg> So lmcdasm, we are ok but mskalski and the OPNFV build system gets hit by this. What are the commonalities/differences?
14:09:58 <lmcdasm> are we sure the Build system is hit?
14:10:05 <stefan_berg> I'm on Ubuntu 14.04 and Docker 1.5.0.
14:10:12 <lmcdasm> i havent tested the latest ISO from Fetig
14:10:25 <stefan_berg> Well, the first official candidate didn't seem healthy right?
14:10:27 <lmcdasm> i know that RC0 ISO was bad, but i dont think its related cause it was bad for me as well across
14:10:35 <stefan_berg> But agreed, we should test a new nightly.
14:10:46 <lmcdasm> but i thought we have fixed that and anything i have dont single Apr 8th (new ISOs) seems fine to me
14:10:47 <[1]JonasB> lmcdasm: RC0 was defenetly broken in the way mskalski describes
14:11:08 <lmcdasm> ok - again - i didnt examine cause i was away - i just did a new build when we all started back
14:11:09 <mskalski> I think it is not dependent from environment
14:11:14 <stefan_berg> So the big questions is if *every* build from the OPNFV build system is broken...
14:11:23 <mskalski> this a bug inside debootstrap
14:11:26 <lmcdasm> @SBERG -
14:11:26 <collabot> lmcdasm: Error: "SBERG" is not a valid command.
14:11:44 <lmcdasm> ok.. well.. again
14:11:48 <lmcdasm> im not sure the paths here.
14:11:53 <[1]JonasB> mskalski: You mention there is a patch for this
14:11:58 <mskalski> https://bugs.launchpad.net/ubuntu/+source/debootstrap/+bug/1001131
14:12:08 <stefan_berg> Well, mskalski, if it was a random thing then I should get hit by this at least once in a while and I've done double digit builds without seeing this.
14:12:09 <lmcdasm> my understaning is we need to test the daily from fetih, i will do this today hopefully
14:12:18 <stefan_berg> lmcdasm: Yes, agreed.
14:12:19 <lmcdasm> and then we should try to apply that patch for mslaski
14:12:35 <lmcdasm> but again.. its hard to help when Sberg, me, Szi have been ok with ISO's
14:12:37 <stefan_berg> And mskalski, yes, that bug id is a perfect match!
14:12:41 <lmcdasm> alot of diff HW
14:12:49 <[1]JonasB> mskalski: What would it take to deploy that patch?
14:12:52 <lmcdasm> not that i dont wanna help im just not seeing the same problem / cant recreate it
14:13:51 <mskalski> stefan_berg: but you can simply reproduce this bug by changing orderning in Packages file, you can event take Packages file from broken ISO and bug will appear
14:15:21 <mskalski> I tested this and found that order in Packages files is responsible for that
14:15:27 <lmcdasm> but thats the thing, each time my packages seem in the same order - comparing two different built ISOs
14:15:56 <mskalski> lmcdasm: are you sure?
14:16:07 <lmcdasm> loking deeper now
14:16:18 <lmcdasm> cause the atp-ftp-archive-release and -deb dont change each build
14:16:19 <mskalski> I compared whit http://artifacts.opnfv.org/genesis/fuel/opnfv-32.iso
14:16:31 <lmcdasm> ok.. but i mean i thought we all said that is dead
14:16:33 <mskalski> and are different
14:16:35 <lmcdasm> im using new builds
14:16:55 <lmcdasm> from apr8 forward.. maybe im missing something - are we trying to fix that ISO specifically ?
14:17:10 <[1]JonasB> From the bug link mskalski provided:
14:17:12 <[1]JonasB> It didn't appear for me in the last month
14:17:12 <[1]JonasB> While today it happens, ooooooooooooooooooooooooooooh.
14:17:12 <[1]JonasB> I have no idea what makes it disappear and what make it appear again
14:17:13 <mskalski> lmcdasm: I also use new builds, they doen't work because of bad order
14:17:45 <mskalski> lmcdasm: this bug is not reproducable always
14:17:51 <lmcdasm> we know that there is transient nature of FUel upstream, so there is some to be expected, but again, each time i build - i dont do a full one - once i have a stable fuel ISO i only do builds that dont redo the whole FUEL ISO
14:18:24 <lmcdasm> so im wondering if there is two issues here again - one is the transient nature of Fuel repo (as we know and are working on) combined with a env/bug that affects your setup to...
14:18:31 <lmcdasm> all guesses though
14:19:40 <[1]JonasB> Ok, we know we have a problem, and I would say mskalski nailed what causes the behavior, although we have no idea on what makes it appear.
14:19:51 <mskalski> lmcdasm: it is not only my setup, this common setup (virtualbox, esxi) so event if now only for me this bug is visible it can back when others start to use genesis
14:20:43 <[1]JonasB> Should we not instead now try to nail the root-cause. I.e. fixing debootstrap. How do we do that?
14:21:02 <mskalski> [1]JonasB: i will try prepare patch to debootstrap
14:21:07 <lmcdasm> well.. ok.. i have 8 ESXI node - not a single issue with any builds past apr8
14:21:20 <lmcdasm> i have two sets of HW blades (sun fix 4x4170 and HP ) and no issues with that
14:21:42 <mskalski> lmcdasm: you said you also do not build fuel from scratch
14:21:44 <lmcdasm> the RC0 build was bad, agreed, but anything since we had ourdiscussion and the test build that i did and even stefan yesterday is working
14:21:47 <[1]JonasB> lmcdasm: again, why debate this, it is enough that there is one instace that fails.
14:21:59 <lmcdasm> fair enough.
14:22:15 <lmcdasm> im just not clear what we are trying to fix
14:22:23 <lmcdasm> and ISO that we know was bad (i.e RC0)
14:22:42 <lmcdasm> or a debbootstrep bug that is known yes, but that 1/3 people building arent seeing the past days
14:23:07 <mskalski> lmcdasm: becacouse this is nature of this bug I assume
14:23:13 <[1]JonasB> lmcdasm: mskalski .iso built at the same time as Stefan built also failed.
14:23:30 <lmcdasm> so you have 2/3 working biulds
14:23:52 <[1]JonasB> mskalski: How long time do you think it would take to apply the patch?
14:23:52 <lmcdasm> again - let me know what you wanna do to address it
14:24:59 <stefan_berg> I'd like to nail down whether it's build or deploy related. I'm still betting on the former.
14:25:42 <stefan_berg> When I was iron out the quirks in the initial build system commits I spun up an Ubuntu instance on EC2 to have a known common environment. Maybe an idea to buy some cycles there and try a build.
14:26:43 <[1]JonasB> stefan_berg: or we just fire up a new build on our build server?
14:26:45 <lmcdasm> we have lots of HW now, we can make all sorts of build setups.. is there something about EC2 that helps? (i dont use em)
14:27:05 <lmcdasm> #agreed. Fetih said we need to do a build anyway before he can install jenkins
14:27:09 <stefan_berg> lmcdasm: Just to have a common reference setup to relate to.
14:27:38 <mskalski> can we at least aggree that problem is inside order of /var/www/nailgun/ubuntu/x86_64/dists/precise/main/binary-amd64/Packages, I tested this and confirmed and described how it can be reproduced
14:27:45 <lmcdasm> #Sberg - ok - well if we are gonna do that, i would say that we make 1 VM that is the BUILD-VM and OVA/QCOW it so that everyone uses the same
14:27:54 <stefan_berg> [1]JonasB: Yes, absolutely, if we have issues with the OPNFV builds then that has top prio.
14:28:04 <lmcdasm> I mean - if we are going to do it to maek a common environment, lets make sure its common for everytne
14:28:19 <[1]JonasB> lmcdasm: Can you fire up a build on Fatihs build server
14:28:29 <fdegir> #info New Ericsson build server has been connected to OPNFV Jenkins
14:28:34 <fdegir> #info builds work fine
14:28:38 <lmcdasm> :)
14:28:41 <lmcdasm> already done
14:28:55 <[1]JonasB> fdegir: Yes but it doesnt install
14:29:02 <lmcdasm> #Fetih  - is there a ISO ready i can test?
14:29:05 <fdegir> yes
14:29:18 <fdegir> http://artifacts.opnfv.org/genesis/fuel/opnfv-40.iso
14:29:30 <lmcdasm> ok.. i iwll spin that up on BL11 today
14:29:34 <fdegir> but we are building on ericsson machine using ericsson contributed build script
14:29:40 <[1]JonasB> fdegir: built when
14:30:17 <fdegir> [1]JonasB: about 3 hours ago
14:30:23 <fdegir> on new build server
14:30:59 <[1]JonasB> fdegir, lmcdasm: lets go with that one, lmcdasm can you install it?
14:31:07 <mskalski> again if this ISO will contain proper order inside Packages files it will deploy, but that does not mean that next bulid will have proper order
14:31:41 <[1]JonasB> #info sdo this is what we do:
14:31:43 <lmcdasm> @agreed- so -0 add a sort into the lists prior to that the lists are always static ro in the alpha order you need
14:31:43 <collabot> lmcdasm: Error: "agreed-" is not a valid command.
14:31:47 <lmcdasm> agreed- so -0 add a sort into the lists prior to that the lists are always static ro in the alpha order you need
14:32:21 <[1]JonasB> #info mskalski Works on a patch for the sequencing/debootstrap issue.
14:33:09 <[1]JonasB> #info lmcdasm Try to install an image built on the old build machine (the one that produced RC0)
14:33:18 <[1]JonasB> Is that OK?
14:34:00 <[1]JonasB> Ok lets move forward
14:34:09 <[1]JonasB> #topic Lab status
14:34:15 <lmcdasm> uh
14:34:18 <lmcdasm> sorry what?
14:34:47 <lmcdasm> we arent testing the new ISO on the new build machine, you want me to test the 32.iso
14:34:47 <[1]JonasB> lmcdasm: You had a lot of updates on the lab
14:34:47 <lmcdasm> ?>
14:35:04 <lmcdasm> yes.. i sent out a mail
14:35:27 <[1]JonasB> Great so we have how many nested envs.
14:35:38 <lmcdasm> 6
14:35:42 <lmcdasm> by EOB today
14:36:11 <[1]JonasB> Great, can we have a fuel env deployed on at least a few of them?
14:36:33 <[1]JonasB> With ODL running
14:36:41 <lmcdasm> so.. each envionment has the FUEL ISO (from Stefan yesterday). with aA compute and Control running and ODL on it
14:36:47 <lmcdasm> lus the VR.. so its all set as a kit
14:36:50 <lmcdasm> all 6
14:37:05 <lmcdasm> the network diagram and one pager to get into aeach lab will come as soon as i have time to finish the doc.
14:37:07 <[1]JonasB> Splended Daniel!
14:37:28 <lmcdasm> as well though - now im confused on the last point, cause i was testing the latest builds - but if i understand im to take one of the blades (onf of 6)
14:37:36 <lmcdasm> and do an old build from that old RC0/Bad ISO?
14:38:04 <[1]JonasB> #info we will have 6 nested fuel env on the Ericsson Lab, by end of to day with descriptions/diagrams and the whole lot!
14:38:21 <lmcdasm> #info  the diagrams may be a bit later
14:38:23 <lmcdasm> :)
14:38:39 <[1]JonasB> So should we after that commision the low half of the lab and reprovision it?
14:38:56 <lmcdasm> need info on what the status is - once the Build server is 100% we will take back blade 1
14:38:57 <[1]JonasB> decomission it should be:-)
14:39:11 <lmcdasm> i was going to do blade1 - blade 6 as a HA (3Con/ 3Comp) setup
14:39:29 <lmcdasm> and then we have two blades for eeryone to have a Build machine on (i.e as many standardized build VMs as you want)
14:39:31 <lmcdasm> for each "user"
14:39:38 <lmcdasm> that was an idea anyway
14:39:44 <[1]JonasB> #info Daniel has prepared a new build server for Fathi
14:39:53 <mskalski> iso: http://artifacts.opnfv.org/genesis/fuel/opnfv-40.iso is ok, you don't need to deploy, just mount and see that base-file is before base-passwd  ubuntu/dists/precise/main/binary-amd64/Packages
14:40:46 <lmcdasm> hmm. well - ok - again, i think its a questino of focus, im working on deploying that ISO so we "have a verified" end to end object fro our build server
14:41:11 <[1]JonasB> #info once Fatih has moved the jenkins jobs to the new buildserver we will rebuild blade 1-6 for bare metal deployments.
14:41:16 <lmcdasm> the bug you are chasing we know, accept, and you are going to fix either in the deb boot strep patch, or do some magic in the list ordering of the install.sh so that the list is always static (or in an order that works)
14:42:00 <[1]JonasB> Anything more on the lab
14:42:23 <fdegir> #info builds have been moved to new server
14:42:37 <fdegir> #info the old build server can be taken out
14:42:48 <lmcdasm> #info - thx Fetih, i will tear it down tomorrow
14:42:57 <lmcdasm> can you turn off anything that isnt needed (services wise)?
14:43:18 <fdegir> what you mean "turn off"?
14:43:24 <fdegir> I disconnected it from Jenkins
14:43:32 <fdegir> I can stop dockerservice etc
14:44:24 <fdegir> lmcdasm: will look into it
14:44:27 <[1]JonasB> Lets move on
14:44:35 <lmcdasm> ok... disconnected is what i meant
14:44:39 <[1]JonasB> #topic autodeploy
14:44:42 <lmcdasm> so i dont break anything upstream
14:44:51 <[1]JonasB> Stefan - updates:-)
14:45:17 <stefan_berg> So I'm working on a prototype to experiment with the DEA concept and autodeploy with libvirt.
14:45:38 <[1]JonasB> Pound info
14:46:07 <stefan_berg> I did a first commit of this today, with the "create_dea" part that takes an existing environment and turns that into a "dea.yaml" (an example is in the commit).
14:46:12 <stefan_berg> #I'm working on a prototype to experiment with the DEA concept and autodeploy with  libvirt.
14:46:19 <stefan_berg> #info Stefan is working on a prototype to experiment with the DEA concept and autodeploy with  libvirt.
14:46:22 <stefan_berg> :)
14:46:54 <[1]JonasB> So how far have you come, does it deploy yet:)
14:47:11 <stefan_berg> Now I'm using that for a deploy.sh "deploy.sh <deafile>" and this is coming along quite nicely, I am able to create environment, define settings and network, start up virt hosts and assign roles to them.
14:47:38 <stefan_berg> What's remaining in this step is to setup the network roles on the interfaces and then deploy, will hopefully get there today.
14:48:00 <[1]JonasB> Should not disturb you then:-)
14:48:06 <stefan_berg> Then I need to fix an outer loop to automate the Fuel master install.
14:48:14 <stefan_berg> Shouldn't be that hard.
14:48:53 <[1]JonasB> Ok anything more for today?
14:48:59 <stefan_berg> Will provide libvirt XML definitions of my environment as a template, a script to automatically define VMs will come a little later (thinking: based on queries to questions about RAM, disk etc).
14:49:03 <lmcdasm> #info - autodeploy sent -sample deploy script for Ostack around
14:49:05 * stefan_berg will shut up now
14:49:24 <lmcdasm> #info - autdeploy - esxi nested worked (create VMs, boot from ISO, setup networks)
14:49:34 <[1]JonasB> Thx
14:49:37 <lmcdasm> #info - autdeploy - will hook into the Fuel Master loop when stefan has it ready
14:49:58 <[1]JonasB> #topic Docs
14:50:11 <[1]JonasB> #Info Jonas is working with docs
14:50:15 <[1]JonasB> Really fun!
14:50:38 <[1]JonasB> So if nothing else we can stop now?
14:51:11 <[1]JonasB> #endmeeting