15:00:24 <fzhadaev> #startmeeting Fuel@OPNFV
15:00:24 <collabot`> Meeting started Thu Dec  8 15:00:24 2016 UTC.  The chair is fzhadaev. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:24 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:00:24 <collabot`> The meeting name has been set to 'fuel_opnfv'
15:00:47 <fzhadaev> #link https://mirantis.webex.com/mirantis/j.php?MTID=m83795240f241e1b1296b6021f19d25f8
15:00:57 <fzhadaev> #info Fedor Zhadaev
15:03:57 <david_chou> #info David Chou
15:23:59 <Greg_E_> ferenc can you let us know the update via chat
15:24:12 <Greg_E_> sorry your voice connection is bad today
15:25:47 <ethfci> danube with boron sr-1 experiencing neutron errors in mechanism driver
15:26:18 <Greg_E_> ODL guy is here at the summit
15:26:23 <ethfci> no topology model can be seen in opendaylight
15:26:29 <Greg_E_> he told me to let him know
15:26:44 <Greg_E_> can you copy/paste error into the chat
15:27:01 <Greg_E_> If I remember ODL has 2 neutron models
15:27:05 <Greg_E_> the old one
15:27:09 <Greg_E_> and new one
15:27:12 <ethfci> i will email it tomorrow, now iam @ home
15:27:19 <Greg_E_> which model
15:27:53 <ethfci> v1 mechanism driver  as i remembered
15:29:28 <Greg_E_> so the older one. which is supposed to work
15:29:59 <ethfci> yes
15:30:18 <Greg_E_> please send the error to me. I’ll try the community folks and our internal ODL team
15:30:32 <ethfci> sure
15:30:45 <Greg_E_> so if I understand correctly this works in colorado 3
15:30:55 <Greg_E_> but is broken in danube?
15:31:35 <Greg_E_> Igor Z. are you online?
15:31:41 <ethfci> when i try with boron sr-1 something is not ok with the communication
15:32:12 <ethfci> when usin Mihail Polenchuk plugin
15:33:26 <Greg_E_> Folks we agreed on a new process to manage Colorado bugs post 3
15:33:38 <Greg_E_> there will be no new releases for Colorado
15:36:44 <OPNFV-Gerrit-Bot> Michael Polenchuk proposed fuel: Turn kvm-plugin back on  https://gerrit.opnfv.org/gerrit/24547
15:40:06 <Greg_E_> for reported but unreleased Colorado issues, please fill out a work around field
15:40:37 <Greg_E_> expected that C3 will be continued to be used and tested by various interested parties
15:40:43 <Greg_E_> until D 1 is out
16:58:43 <DanSmithEricsson> Hello - Stefan ar du dar?
16:58:47 <DanSmithEricsson> Or Fatih?
17:33:39 <DanSmithEricsson> Any other fuelers out there?
17:37:22 <NikoHermannsEric> DanSmithEricsson: :-D
17:37:41 <NikoHermannsEric> but only 20 more seconds
19:47:59 <OPNFV-Gerrit-Bot> Michael Polenchuk proposed fuel: Point to latest repos of fuel 10/newton  https://gerrit.opnfv.org/gerrit/25689
22:45:47 <AlexAvadanii> sergmelikyan: hi! Cristina told me you had some issues with booting Fuel off an USB stick - maybe this helps https://ubuntuforums.org/showthread.php?t=1980215
22:46:58 <AlexAvadanii> or a better resource http://www.syslinux.org/wiki/index.php?title=Isohybrid
22:49:27 <sergmelikyan> Thank you!
22:54:13 <sergmelikyan> doesn't answer question how to take existing iso and make it bootable :(
22:54:23 <sergmelikyan> i am already struggling 6h with this shit :(
05:27:27 <OPNFV-Gerrit-Bot> Merged fuel: Point to latest repos of fuel 10/newton  https://gerrit.opnfv.org/gerrit/25689
05:31:12 <OPNFV-Gerrit-Bot> Merged fuel: Turn kvm-plugin back on  https://gerrit.opnfv.org/gerrit/24547
15:05:33 <AlexAvadanii> sergmeli_: hi! I see youdon't have a JIRA account, or I would have added you as a watcher to this bug report I just created: https://jira.opnfv.org/browse/FUEL-236
15:05:46 <sergmeli_> I have )
15:06:31 <sergmeli_> assigned bug to me - thank you!
15:06:38 <AlexAvadanii> sergmeli_: ah, cool! are you at the plugfest venue?
15:06:42 <sergmelikyan> yeah :)
15:13:07 <mskalski> https://bugs.launchpad.net/fuel/+bug/1648741 that moment when you realise that creating local mirrors was a good decision
15:22:40 <AlexAvadanii> sergmeli_: I added a comment in FUEL-236 about ISO handling in Fuel@OPNFv
15:25:58 <sergmeli_> AlexAvadanii: thx! I've rebuilt my installation drive - let's see if it's works ;)
13:59:29 <DanSmithEricsson> Hello Fuelers.. so I have spent a good part of the weekend trying to bring up the CI on the colorado release 3.0 in the OPNFV site
14:00:01 <DanSmithEricsson> and its bad.. every single deployment on the last 11 days ISOs fail.. there seemst o have been some new "tools packages" that have been added in the installer that complain at the enbd
14:00:35 <DanSmithEricsson> if there is anyone around in the project, happening to read this and knows if this release was actually tested by hand by anyone- please reach out and let m eknow
15:12:27 <mskalski> DanSmithEricsson: https://bugs.launchpad.net/fuel/+bug/1648732
15:13:09 <mskalski> DanSmithEricsson: also I responded to your email on opnfv tech discussion list
15:58:01 <DanSmithEricsson> hey Michal.. reading..
15:58:55 <DanSmithEricsson> thanks for the information - question though  - how come the CI is saying things are passing then? did you do a hotfix on those systems?
16:00:26 <DanSmithEricsson> at any rate, seems like there is a workaround (good!). the guys (fatih, Jose, Niko) can implement this in the CI setup tomorrow if they need.
16:00:46 <DanSmithEricsson> still strange why the CI says everything is "good" - i dont see where this was added to DEA/DHA.
16:00:50 <DanSmithEricsson> anyway .. thanks
16:06:03 <mskalski> DanSmithEricsson: you mean our stable/colorado  builds? we are using local mirros there https://git.opnfv.org/fuel/tree/deploy/config/dea_base.yaml?h=stable/colorado#n633
16:06:33 <mskalski> DanSmithEricsson: changes in upstream ubuntu repos are not reflected there
16:51:58 <DanSmithEricsson> hey Michal
16:53:02 <DanSmithEricsson> ok..so the bug only comes into play when people use the manual installation / instructions (and external repos?).. im confused, if the build is from the FUEL build (the systemd package is in the FUEL itself) then how does using external report vs. internal Miorrors come nto play?
16:53:18 <DanSmithEricsson> or do you mean - that the internal mirrors is that we are not using a 'fresh build" of upstream fuel?
17:05:34 <mskalski> DanSmithEricsson: source of the problem is a change in upstream ubuntu repository, systemd package was moved to updates, when you use upstream ubuntu repository (by default for manual installation) this package will be installed and will confuse puppet how to start services, which you see when it is trying to start atop
17:06:07 <mskalski> DanSmithEricsson: when you use local mirros systemd package will not be installed, it is not present in local mirror
17:08:33 <mskalski> by upstream ubuntu repositories I mean archive.ubuntu.com, not fuel mirantis repos
17:10:15 <DanSmithEricsson> ok.. thanks..
17:10:24 <DanSmithEricsson> so the issue is with the Ubuntu image that is pulled down from upstream.
17:10:29 <DanSmithEricsson> not with upstream FUEL i meant
17:24:08 <mskalski> we build installation image during deployment, one of the source of packages for this image is trusty-updates, before above change systemd was not there and installation image did not include this package
17:24:40 <mskalski> no it is installed as a dependency
17:24:44 <mskalski> *now
17:25:15 <mskalski> the other workaround is to define that we want to have systemd-shim package in our installation image
17:25:35 <mskalski> this package is in conflict with systemd and prevent from installing it
17:25:40 <DanSmithEricsson> ok.
17:25:47 <DanSmithEricsson> any reason why FUEL dies afterwards then?
17:26:14 <DanSmithEricsson> seems like when the deployment is done.. and i try to use FUEL afterwards.. its dead..  (maybe a different issues - just checking that they "shouldnt" be related).
17:27:21 <mskalski> what do you mean by dead?
17:27:45 <DanSmithEricsson> well.. as soon as deployment is done.. the UI becomes unresponsive.. "fuel xyz" command hang..
17:27:51 <DanSmithEricsson> (im looking more now)
17:30:01 <mskalski> sounds like nailgun is not up
17:30:23 <DanSmithEricsson> ya..  thing is each time i do a deploy, i have to reinstall FUEL
17:30:31 <mskalski> or nginx which proxy request to nailgun
17:30:56 <DanSmithEricsson> anyway.. i think for today (and to get the rest of the lab "working") ill go to Colorado 2.0 - then the guys can update / upgrade in the CI as it work
17:31:05 <DanSmithEricsson> thanks alot for the help!
17:32:50 <DanSmithEricsson> Hey Michal
17:32:54 <DanSmithEricsson> something doesnt make sense in what you said
17:32:58 <mskalski> this problem will be visible for all colorado releases, actually for all fuel releases which use 14.04
17:33:01 <DanSmithEricsson> cause in FUEL. i have a new directoty atp
17:33:02 <DanSmithEricsson> atp*
17:33:15 <DanSmithEricsson> atop*.. and in FUEL 2.0 COlorado this directoyy in /var/log on FUEL doesnt exist
17:33:24 <DanSmithEricsson> so there is something about "ATOP" that is no in FUEL (not just Ubuntu)
17:33:46 <DanSmithEricsson> if you would like to take a look at the deployed / dead system you are more than welcome
17:34:25 <DanSmithEricsson> but 4 deploys, and they failed (with the externals), btu FUEL should not be dead afterwards i dont think (form the info you outlined). so there is something else going on i think.. maybe its local to these machines, but i doubt it
17:35:10 <mskalski> atop is just first services which is started by puppet during deployment, there is nothing specific with this services
17:35:30 <DanSmithEricsson> hmm. ok - but its been introduced in FUEL in Col. 3.0 ?
17:35:33 <mskalski> problem is that puppet is trying to use systemctl to start it
17:35:40 <DanSmithEricsson> on the FUEL node?
17:35:54 <DanSmithEricsson> im sorry - super confused, i thought this was only to the Ubnutu (deployed nodes).
17:36:31 <mskalski> ok we are talking about two different things now
17:36:33 <DanSmithEricsson> but looking on FUEL, i can see that ATOP is there as well, being used (its running all the time by the way ) - so maybe i dont undersatnd, but are we saying that the introduction of ATOP also impacted the systemd called in FUEL
17:36:39 <DanSmithEricsson> ok..  i am talking about my FUEL node
17:36:49 <DanSmithEricsson> on the FUEL node, comparied to FUEL from Col 2.0
17:37:03 <DanSmithEricsson> there is a new ATOP pacakge that wasnt there (i only looked cause of the other conversation).
17:37:15 <DanSmithEricsson> when i do a deployment and it fails.. the FUEL then dies shortly afterwards..
17:39:50 <mskalski> all colorado releases uses the same version of fuel 9.0.1
17:40:02 <DanSmithEricsson> hmm.
17:40:05 <DanSmithEricsson> ok.. cool.
17:40:08 <DanSmithEricsson> well..i dunno then..
17:40:31 <DanSmithEricsson> been messing around with this Col 3.0 release since Thursday on 2 BareMetal and 4 Virtusal Pods and cant get a single deploy to work.
17:40:51 <DanSmithEricsson> so for me :) - i will hand this to the CI guys .. cause the same setup that works with 2.0 doesnt work with 3.0
17:41:11 <DanSmithEricsson> and the repos might  have something to do with it, but then i dont know about the FUEL dieing (that is new)..
17:41:17 <DanSmithEricsson> thanks for all the help Michal!
19:25:47 <DanSmithEricsson> hey there.. i tried the workaround in the link you provided and 3/5 nodes still failed to deploy.
19:25:50 <DanSmithEricsson> just fyi
23:13:59 <DanSmithEricsson> hey Fuelers.. just FYI - it would appear that we cannot install FUEL manually anymore
23:14:38 <DanSmithEricsson> i have tried the workarounds outlined int he bugs, but none of them seem to work.. this means that all userlabs (anything not installed via CI) fails -.. the downside? - while i have lots of backups of everyone's environments. i dont have a workable installation to move your stuff too
23:15:04 <DanSmithEricsson> i will try to back up and move things to a safe location for now.. but sadly the sparewheel comes down tomorrow and without something to install to - we will all be without userlabs
23:15:13 <DanSmithEricsson> ac -p
08:01:15 <pma> <DanSmithEricsson> hello, please try to apply this patch (http://hastebin.com/ifehotefaq.rb) on fuel master node if deployment is still failing
08:21:39 <DanSmithEricsson> hey pma - im not building, but using a pre-built ISO
08:21:54 <DanSmithEricsson> can i still hack that into the manifest on the fuel prior to deploying or doest it have to be built in?
08:22:11 <DanSmithEricsson> cannot i not just add those to the "end" of the initial packages list?
08:23:13 <pma> you could inject that into the manifest on the fuel prior to deploying
08:24:05 <DanSmithEricsson> ok.. im about to do a deploy - question
08:24:11 <DanSmithEricsson> do i still need to use internal mirrors then
08:24:15 <DanSmithEricsson> or just this patch?
08:24:15 <pma> or add this to "initial packages list" if you have fresh install
08:24:24 <DanSmithEricsson> ya.. i have a fresh install
08:24:32 <DanSmithEricsson> so in the settings where we do the Urls
08:24:39 <DanSmithEricsson> add all three, or just systemd you figure?
08:24:52 <DanSmithEricsson> (and the mirrors? keep em you think?).,
08:25:07 <pma> just this patch, it doesn't matter local or external repo is using
08:26:14 <DanSmithEricsson> ok.. ill give it a try right now :)
08:26:20 <pma> Ok
08:28:17 <DanSmithEricsson> <pma> <DanSmithEricsson> hello, please try to apply this patch (http://hastebin.com/ifehotefaq.rb) on fuel master node if deployment is still failing
08:28:19 <DanSmithEricsson> oops
08:28:29 <DanSmithEricsson> http://picbin.org/src/1267
08:28:36 <DanSmithEricsson> like that?
08:30:09 <pma> DanSmithEricsson> yep
08:30:55 <DanSmithEricsson> so.. if we do this on BM pod and reap it
08:30:57 <DanSmithEricsson> will it show up?
08:32:27 <pma> sure
09:02:54 <pma> please review https://gerrit.opnfv.org/gerrit/25779
09:23:24 <DanSmithEricsson> done.. thanks.. that worked!
09:27:28 <pma> cool, but not sure about patch, I mean C3.0 already has been tagged
09:28:21 <pma> will it be 3.1 ? or it's just for CI
09:30:22 <DanSmithEricsson> hmm. i have no idea how that works.
09:30:37 <DanSmithEricsson> for me, for 3.0 i think the "easiest" would be a documentation fix to add that in
09:30:49 <DanSmithEricsson> for CI - it would be picked up by the reaping (and CI seems to be "ok")
09:33:51 <pma> Ok
09:38:01 <mskalski> http://ircbot.wl.linuxfoundation.org/meetings/opnfv-fuel/2016/opnfv-fuel.2016-12-08-15.00.log.txt "15:33:38 <Greg_E_> there will be no new releases for Colorado"
09:39:58 <mskalski> if you use colorado 3.0 ISO http://artifacts.opnfv.org/fuel/colorado/opnfv-colorado.3.0.iso local mirros does not have problematic change included
09:42:58 <mskalski> #endmeeting