17:00:10 #startmeeting Joid weekly 17:00:10 Meeting started Wed Dec 9 17:00:10 2015 UTC. The chair is arturt. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:10 Useful Commands: #action #agreed #help #info #idea #link #topic. 17:00:10 The meeting name has been set to 'joid_weekly' 17:00:54 #info David Duffey 17:01:24 #info Artur Tyloch 17:01:26 Hi arturt 17:01:42 narindergupta: have you started gotomeeting ? 17:02:14 when I am trying to join gotomeeting it shows message meeting in progress 17:02:58 arturt, waiting for you 17:04:14 #info David Blaisonneau 17:04:21 #topic agend bashing 17:04:30 #link https://etherpad.opnfv.org/p/joid 17:07:16 #info Iben Rodriguez 17:08:00 #topic sprint backlog review - MAAS Pilot 17:09:55 #info #link MaaS Pilot location list https://wiki.opnfv.org/pharos/maas_getting_started_guide#maas_pilot 17:15:13 #info * Orange Pod 2 - https://wiki.opnfv.org/opnfv-orange - working to resolve a configuration issue and reinstall 17:15:38 #info * Intel Pod 5,6 https://wiki.opnfv.org/get_started/intel_hosting - eta dec 15 DavidB@Orange 17:15:39 17:17:57 #info * Juniper Pod 1 - Iben working with Samantha@canonical to document the setup 17:19:38 #info 17:19:38 * Dell Pod 2 - https://wiki.opnfv.org/dell_hosting - TPM Mark Wenning @canonical and Vikram @Dell 17:21:34 #info * UTSA - TBD 2016 17:21:42 #info * CENGN - TBD - Nicolas@Canonical 17:21:54 #info * Huawei - TBD Prakash to inform us on status 17:23:19 #topic sprint backlog review - JOID-25 17:23:28 #topic sprint backlog review - JOID-29 17:30:04 #topic sprint backlog review - JOID-25 17:41:15 #info #link JOID charm (plugin) more info : https://wiki.opnfv.org/joid?&#juju_and_maas_training_materials 17:41:15 [08:20am] <— sent to pharos project meeting this morning 17:42:45 #info please review the #link for the pharos meeting minutes http://ircbot.wl.linuxfoundation.org/meetings/opnfv-pharos/2015/opnfv-pharos.2015-12-09-16.00.html for more info on writing charm 17:44:24 https://jujucharms.com/docs/stable/authors-charm-writing 17:46:30 #info added #link Your first charm starts here! to wiki 17:46:53 : #info added #link Your first charm starts here! to wiki https://jujucharms.com/docs/stable/authors-charm-writing 17:47:34 #topic sprint backlog review - JOID-16 17:56:25 #action iben update https://wiki.opnfv.org/joid/get_started with info about Jenkins installation in POD 17:56:40 #info #link https://wiki.opnfv.org/octopus/jenkins_slave_connection how to connect jenkins slave 17:56:56 #action David_Orange update https://wiki.opnfv.org/joid/get_started with info about JOID CI job for a new POD 17:57:57 #topic sprint backlog review - JOID-25 17:58:16 * fdegir connected pod5 to opnfv jenkins 17:59:04 about jenkins installation; please do not install jenkins to any PODs that is going to be used by OPNFV CI 17:59:28 #info https://wiki.opnfv.org/joid/get_started <— updated - please review and advise how to improve 17:59:33 hello fdegir 17:59:40 hi 17:59:46 i will let them know on audio 18:00:01 thanks 18:00:03 I already explained this to narindergupta 18:00:07 so he should be aware of this 18:00:11 k 18:01:10 but it seems pod5 lacks jumphost 18:01:15 that's a problem 18:04:00 fdegir, pod5 has jumphost 18:04:19 narindergupta: the ip I got from you is the ip of the node1 18:04:25 narindergupta: and pod5 has 5 nodes 18:04:33 yeah that is jumphost 18:04:41 as intel has only five servers 18:04:43 narindergupta: but pharos spec says 5 + 1 18:04:59 narindergupta: and all the test projects rely on the jumphost 18:05:04 yeah both intel pods 5 and 6 hs total 5 18:05:16 narindergupta: I noticed that as well 18:05:24 and my deployment uses 3 control and 1 compute 18:05:43 narindergupta: but some test projects probably expect 2 computes 18:05:52 narindergupta: yardstick for example 18:06:59 narindergupta: you have the mail from Ana 18:07:18 narindergupta: and she says: "We need a baremetal system, 3 x controllers, 2x compute, 1x jump host." 18:08:11 fdegir, ok it is something intel has to provide. 18:08:26 narindergupta: I reported this to them already 18:08:41 fdegir, ok thanks 18:16:27 narindergupta: until this is resolved 18:16:27 narindergupta: we can start creating jenkins jobs for joid 18:17:08 fdegir, David from Orange is helping us on that. And i believe he is working with you on the same. 18:17:25 narindergupta: what he does it for testing purposes 18:17:48 narindergupta: and he uses stuff (scripts, etc.) that is locally available on their pod 18:17:56 narindergupta: which the CI doesn't allow 18:18:07 narindergupta: everything must come from your OPNFV git repo 18:18:15 narindergupta: deployment, cleanup, and so on 18:18:38 narindergupta: so David's job is not a valid job for CI 18:24:16 fdegir, everything we have in joid git i think once we have valid script 18:24:27 then we are planning to include those in repo 18:24:40 all scripts are part of ci and git 18:42:05 narindergupta: so it is not possible to create CI stuff as of yet 18:44:26 fdegir, problem is i do not know what to do for CI and we asked help from David. If some one can lead me with examples then i think of how to automate? 18:46:08 narindergupta: all the other installers do this already 18:46:13 narindergupta: they're the examples 18:46:43 narindergupta: you should have a ci directory in your repo and place the scripts that are supposed to be executed by CI in this directory 18:46:53 will you point me exact scripts which does that? And it should be on jump host or somewhere else 18:47:08 fdegir, i have the same way 18:47:16 under joid i have ci directory 18:47:24 which has script to do deployment 18:47:31 like deploy.sh and clean.sh 18:47:35 narindergupta: then what did you mean with "we are planning to include those in repo" 18:47:44 narindergupta: and "once we have valid script" 18:48:16 fdegir, yeah David is working on scripts based on joid and once we have those we can include in joid thats what i meant 18:48:27 narindergupta: so they're not there yet 18:48:56 narindergupta: well, I see big risk here 18:48:58 fdegir, yes for that once David done the completion i have plan to include those specific ones 18:49:05 narindergupta: we can't even deploy stuff from jenkins 18:49:12 narindergupta: test projects are in the pipe 18:49:20 narindergupta: and we don't even have the jumphost 18:49:39 fdegir, i am more confused now. 18:49:52 fdegir, earlier you mentioned that we do not need jenkins 18:50:29 I said you do not need Jenkins installation on your pod 18:50:57 fdegir, so what i need to ci then? 18:51:10 you need to have automatic deployment scripts for CI 18:51:22 once you have those scripts, you also need jenkins jobs to execute those scripts 18:51:37 on jumphost 18:51:41 fdegir, we have scripts already to do deployment 18:51:53 david is working on jenkins scripts 18:51:59 then what are the scripts you are waiting for David to fix? 18:51:59 and document the same 18:52:08 jenkins script 18:52:59 git clone http://gerrit.opnfv.org/gerrit/joid.git is our joid code 18:53:23 but David is using everything from local copy 18:53:35 so noone can guarantee it will work 18:53:49 for CI 18:54:18 it can work on Orange POD - that's OK if they want to go that way 18:54:50 but it won't work in CI since all the paths in the scripts created by David points to the home directory of whatever user 18:59:53 basically we are integrating th ejoid to labs which are participating and putting config files in each lab directory and as part of joid itself 19:00:33 so you can git clone the joid and then run it from joid/ci and deploy.sh copies to joid/ci and run deployment everything from there 19:01:41 when I look at David's patch, I see lots of things there 19:02:17 virsh commands for cleanup, recovery, config, etc 19:02:30 they are provided by projects normally 19:02:33 not by ci 19:05:57 yes those are parts of our reply.sh file 19:06:16 sorry deploy.sh to create a maas and bootstrap vm and automated 19:07:49 I don't know, I'm pretty confused right now 19:07:56 what comes from where, who does what 19:09:13 can u give me one example ci command ci run after clone the git for installer 19:13:21 I can give you two 19:13:25 https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=blob;f=jjb/compass4nfv/compass4nfv.yml 19:13:43 lines 475 to 505 19:14:10 https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=blob;f=jjb/fuel/fuel.yml 19:14:12 line 321 19:14:16 which executes 19:15:04 https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=blob;f=jjb/fuel/fuel-deploy.sh 19:15:14 and apex is doing this as well 19:20:28 fdegir, looks like it should be easy as in case of joid ./deploy.sh --dha $CONFDIR/dha.yml --network $CONFDIR/network.yml it should be 19:20:29 narindergupta: still looking at David's work 19:20:34 narindergupta: he has pod config 19:20:46 for orange pod I suppose 19:20:48 do you have pod config for intel pod5? 19:20:59 fdegir, ./deploy.sh -o liberty -s odl -t ha -l intelpod5 19:21:06 yes 19:21:12 where is the config located? 19:21:34 everything under maas/intel/pod5/deployments.yaml 19:22:00 good! 19:22:05 so Maas installer run 02-maasdeploy.sh intelpod5 19:22:18 so i think basic work is already there 19:22:33 please don't misunderstand me btw 19:22:39 just trying to understand where you are 19:22:43 not at all 19:22:46 and how can I help you to come to CI 19:22:50 anyway 19:22:57 as i just want to know if there is risk or gap how to resolve it 19:22:58 this command will do the deployment 19:23:08 ./deploy.sh -o liberty -s odl -t ha -l intelpod5 19:23:19 any pre or post steps like David does? 19:23:26 do we need cleanup? 19:23:32 before issuing above command? 19:23:51 i believe clean up is done before 19:24:11 ok there are two cases here i think 19:24:34 first is fresh install with first time install. 19:25:02 then run ./02-maas-deployment.sh intelpod5 this is create maas and bootstrap vm 19:25:29 then run ./deploy.sh -o liberty -s odl -t ha -l intelpod5 will do the deployment 19:25:46 if its already deployed once then ./clean.sh and ./deploy.sh -o liberty -s odl -t ha -l intelpod5 will deploy it 19:26:04 I think other installers do fresh install 19:26:48 if need fresh installs then run ./02-maas-install.sh intelpod5 again and then ./deploy.sh -o liberty -s odl -t ha -l intelpod5 19:27:20 i have method for both 19:28:19 installer does not have to re created again and again 19:28:53 and our ./clean.sh will clean all the environment except installer vm 19:29:45 ok 19:29:56 so if one wants to clean the installer vm as well? 19:30:29 which is what is done by other installers if I'm not totally mistaken 19:31:07 that's why David has that joid-destroy script then 19:31:17 so the process on CI will be 19:31:21 -destroy-vm 19:31:28 -02-maas-install 19:31:31 -deploy.sh 19:32:03 no need to destroy VM 19:32:11 mass-install will destroy iy 19:32:12 it 19:32:22 just run 02-maasdeploy.sh 19:32:24 but what if we do in order to make the ci processes same for all the installers 19:32:44 sure no issue then run ./clean.sh 19:32:46 we always do fresh install - including creation of VMs 19:32:49 destroy the vm 19:33:02 and then run 02-maas-deploy,sh 19:33:07 then deploy.sh 19:33:12 ok 19:33:30 I'll do this quickly, probably tomorrow 19:33:30 and come back to you 19:33:37 do you use pod5 for any reason? 19:33:45 it'll become ci pod meaning hands off 19:34:03 sure no problem. I think David is working on modifying other within config as well. 19:34:17 ok 19:34:21 like ceph-monitor password etc.. 19:34:34 so that same won't be used for security purpose 19:34:43 other? 19:34:48 you mean orange pod? 19:35:37 anyway, thanks for the info, this was really helpful 19:35:50 will send you the logs when I have them 19:42:06 fdegir, i think we doing for orange which can excerize on all labs 19:42:37 other configuration parameter within openstack 19:42:53 like passwords and any other secret information. 19:46:00 narindergupta: we have a new repo which is not readable to the world 19:46:13 we plan to move this type of sensitive information to that repo 19:46:27 ok 19:46:28 but before doing this 19:46:43 do I need all this secret stuff fixed for intel pod5 19:46:48 or will it just work? 19:46:48 in that case it needs to segregate from JOID pretty soon? 19:47:00 no not at all we have default values 19:47:06 ok, good 19:47:07 it will just work 19:47:12 about the secure repo 19:47:25 you're part of the discussion 19:47:25 Frank, Jonas, Tim, Weidong 19:47:25 yeah 19:47:32 once something comes out of that discussion 19:47:39 that something should end up in this new repo 19:47:40 about the config file you meant? 19:47:43 yes 19:47:52 all this type of password, ip, mac, etc. etc. 19:48:10 deployment config, pod config, and so on 19:48:20 yeah we will taking those and replacing in installer defined repo 19:48:39 yep 19:48:43 got you i see where we are going and make sense 19:48:58 until that happens, we have to use what we have 19:49:02 and switch to them once they're settled 19:49:29 I have to drop now 19:49:33 yeah no problem as once settle then installer needs to write parse engine and modify accordingly 19:49:38 sure please 19:49:44 talk to you tomorrow 20:08:28 narindergupta: https://gerrit.opnfv.org/gerrit/#/c/4161/ 20:11:51 fdegir, looks like you had the template. and modified the one and everything looks good to me. Were yo able to run an dverufy 20:12:40 narindergupta: will run it shortly 20:12:44 once a releng committer gives +2 20:12:52 ok 20:13:20 joid there is a change 20:14:10 02-maasdeploy.sh intelpod5 is correct command 20:14:19 ./02-maasdeploy.sh intelpod5 20:15:33 narindergupta: thanks, fixed that 20:15:38 please have a look at that part again 20:17:30 yes looks good now lets verify it if issue exist then i am happy to assist 20:18:01 may i know one moe thing this builders: tag is for deployment right 20:18:13 as joid do not have any build script to follow 20:19:07 our charms do it run time if needed 20:20:19 tag? 20:23:37 there is option builders: under which we have deploy shell commands 20:27:46 yes 20:27:49 you don't have to have build 20:27:55 it is jenkins term 20:27:59 it calls everything as build 20:28:10 ok 20:40:02 narindergupta: it failed 20:40:25 narindergupta: https://build.opnfv.org/ci/view/joid/job/joid-daily-deploy-master/1/consoleFull 20:43:12 fdegir, yeah this portion i added today but could not verify 20:43:45 will work on it basically this will create cloud/openrc file which contains the file to login into openstack 20:44:33 ok 20:44:50 let me know when new version of the script is in the repo so I can give it a try 20:47:54 sure 20:48:03 fdegir, also do you know why opnfvsdn=''\''odl'\''' 20:48:15 is that way i thought it should be clean old only 20:48:18 old only 20:48:44 are you talking about deploy script or? 20:49:42 ok 20:49:43 got it 20:49:47 it was a problem in my part 20:49:51 and fixed that already 20:50:07 narindergupta: check this log please 20:50:08 https://build.opnfv.org/ci/view/joid/job/joid-daily-deploy-master/2/consoleFull 20:50:26 quotes are not there anyomre 20:50:27 opnfvsdn=odl 21:21:05 fdegir, thanks and i have checked in the fixes 21:21:12 will you please retry it now? 21:22:29 fdegir, also i am not seeing output of ./02-maasdeply.sh file? Was that ran successfully? 21:22:31 narindergupta: failed with different error 21:22:31 https://build.opnfv.org/ci/view/joid/job/joid-daily-deploy-master/3/console 21:23:13 narindergupta: could this be the reason? 21:23:15 sudo: no tty present and no askpass program specified 21:23:25 jenkins user might need passwordless sudo 21:24:02 yes jenmkins needs passwordless sudo 21:24:10 i can edit the profile 21:24:26 or you can 21:25:25 this one need maasdeploy script first 21:25:27 can you do that please? 21:29:05 yeah looking 21:36:59 fdegir, i fixed that but you need to run ./02-maasdeploy.sh intelpod5 as well for getting the API file 21:37:04 narindergupta: my power was out earlier, and I need to know how to bring the nodes and services back online - a good test. I have used the maas UI to power them controller and compute nodes back on, but I can't SSH to them and few of the services seem to be active per the status command 21:37:58 narindergupta: that script is there already 21:38:06 fdegir, ok 21:38:10 running it now? 21:38:18 bryan_att, sure 21:38:32 narindergupta: how long should I have to wait, and is there any manual action I need to take? 21:38:40 bryan_att, first you need to bring the jump host server 21:38:48 narindergupta: it seems this is better 21:38:49 * fdegir https://build.opnfv.org/ci/job/joid-daily-deploy-master/4/console 21:39:01 narindergupta: and for my manually created congress container, how do I start it again? 21:39:27 narindergupta: jumphost was powered on and maas/juju UIs are active 21:39:29 just check whether container is already started or not 21:39:44 use sudo lxc-ls 21:39:54 ok 21:40:16 returns nothing 21:40:29 (from the jumphost) 21:40:56 i think owe did install that on control nodes correct? 21:41:06 then juju ssh nodes-api/0 21:41:15 yes, on the control node 21:41:28 now run sudo lxc-ls 21:41:37 and check your container 21:41:49 https://www.irccloud.com/pastebin/dAqa0LAV/ 21:41:57 some error 21:42:31 I can 21:42:36 hun i am hoping you used the same user during install? 21:42:46 (can't) SSH to it 21:42:51 Yes, unbuntu 21:43:10 (ubuntu) 21:43:18 so can you shutdown down your control and compute node and bring up only control node 21:43:36 OK, via maas? 21:44:12 (or manually) 21:44:22 lets try manually 21:44:44 ok, shutting down by soft press on the power switch 21:45:07 ok thanks 21:46:22 i think you should shutdown the bootstrap node lso 21:46:24 also 21:46:52 as per google this happens if ip address gets changed 21:47:13 ok, I will need to connect to IRC on a different machine, hang on 21:47:22 ok 21:47:32 its like powering system on one by one 21:47:39 first jumphsot 21:47:57 make sure both maas and bootstrap nodes are up 21:48:16 then control node and then compute node 21:48:21 ok, powering jumphost back on 21:49:16 ok 21:49:58 how do I know when the maas and bootstrap nodes are ready, is it when the maas and juju UIs are working? 21:50:09 correct 21:52:28 do I need to poweron the bootstrap node via maas? That's what I did last time. 21:58:17 narindergupta: the juju tab is spinning ("connecting to the juju environment"). Seems I need to power the bootstrap node on by MAAS, right? 21:58:18 no bootstrap node is marked as auto power on in lvm itself 21:58:32 if not then yes 21:59:00 seems not, so I will start it via maas 21:59:07 ok yeah then 21:59:31 now try to power on control node first and wait 22:00:29 fdegir, good news maas deployment is completed 22:00:57 fdegir, now deploy.sh script should be running? 22:04:18 ok, JuJu UI is back. 22:08:17 narindergupta: what's next? 22:08:37 cool now startup control node 22:08:58 and run juju status --format=tabular 22:09:03 for observation 22:09:23 after say 2 3 minute you can start the compute node 22:09:41 and check juju status --format=tabular 22:10:19 narindergupta: deploy.sh failed 22:10:44 fdegir, whats the error? 22:10:45 and that's all from me for today 22:10:56 after 2-3 minutes from power on, or some other event in the status output? when I did this before (Arno) I had to wait 2-3 minutes after all OpenStack controller services were back, or the nova-compute would not be registered properly with the ccontroller, and I would have to restart it 22:11:10 narindergupta: https://build.opnfv.org/ci/job/joid-daily-deploy-master/4/console 22:12:06 fdegir, permission denied ok i will have a look 22:16:04 narindergupta: ok, I did "sudo juju status --format=tabular | grep lost" until only the compute node services were listed. I'll power it on now 22:17:04 sure thanks 22:24:24 narindergupta: so now I login to the controller and do the lxc-ls command? 22:24:40 yeah please? 22:25:02 juju ssh nodes-api/0 22:25:11 OK, still getting the error I did before 22:25:26 https://www.irccloud.com/pastebin/cLTmUSqC/ 22:25:53 does juju status --format=tabular looks good? 22:25:58 yes 22:26:21 ok lets try to get ip address of the system and we can try using ip address 22:26:32 ssh ubuntu@ 22:27:42 same error 22:27:44 https://www.irccloud.com/pastebin/CsUq42Bt/ 22:28:01 maybe I forgot the ubuntu 22:28:48 didn't seem to help 22:28:52 https://www.irccloud.com/pastebin/rKzBaWmd/ 22:29:11 do not use juju to ssh 22:29:17 just simple ssh 22:29:37 ok, that didn't work either 22:29:38 like ssh ubuntu@192.168.10.21 22:29:47 same error? 22:29:53 yes, as pasted 22:29:58 can we try compute?. 22:30:07 juju ssh nova-compute/0 22:30:15 Are you able to get to that host at all? I wonder if the first two issues at the following are relevant: http://edoceo.com/notabene/ssh-exchange-identification . 22:30:45 https://www.irccloud.com/pastebin/mzsabOvT/ 22:31:19 can u try without sudo? 22:31:54 https://www.irccloud.com/pastebin/sluGFH0i/ 22:32:42 should I not have using sudo for the juju commands? I thought it was required somehow. 22:33:23 bryan_att, not for juju and i am modifying the ownership in envuronments.yaml so that no sudo needed 22:33:58 OK, is there something I need to do to reset it, or maybe there 22:34:05 s some other issue? 22:36:30 can we run ssh -v 192.1.68.10.21 22:36:46 can we run ssh -v ubuntu192.1.68.10.21 22:36:50 can we run ssh -v ubuntu@192.1.68.10.21 22:37:28 I assume you mean ssh -v ubuntu@192.168.10.21 22:37:58 ok 22:38:20 https://www.irccloud.com/pastebin/syuyxCQl/ 22:40:50 seems something strange in the networking setup. Obviously the maas VM was able to boot the nodes via wake-on-lan. and the containers are working fine, as well as the native services (at least reported fine in the juju status command) 22:41:16 ubuntu@jumphost:~$ sudo juju status --format=tabular 22:41:16 [Services] 22:41:16 NAME STATUS EXPOSED CHARM 22:41:16 ceilometer active false local:trusty/ceilometer-44 22:41:16 ceilometer-agent false local:trusty/ceilometer-agent-20 22:41:16 ceph blocked false cs:trusty/ceph-42 22:41:16 cinder active false cs:trusty/cinder-31 22:41:55 (sorry, wrong paste method!) 22:42:05 https://www.irccloud.com/pastebin/0WS9L4Ip/ 22:43:01 (all OK except for ceph, for some reason) 22:51:06 looks like units can talk to each other but can not access with your keys 22:51:52 i am sure you will have same issue while login in through 192.168.10.113 as well 22:52:13 I can't even ping .21, .22, or .23 22:52:44 .21 and .22 are controller node, .23 is compute 22:53:00 .20 is bootstrap, which I can ssh to 22:53:53 something funky in how the node bridges are setup (or were restored) maybe? 22:54:05 do nslookup compute1.maas 22:54:23 yeah looks like to me 22:55:04 so may be first time it was fine but reboot does not survive and I am thinking may be route is configure there correctly 22:55:22 https://www.irccloud.com/pastebin/EOJZsU15/ 22:55:43 so dos is find 22:55:45 fine 22:55:57 can we check router ip in maas guy configured? 22:56:48 where in the maas gui? 22:56:58 click on cluster tab 22:57:15 on right edit icon click on that 22:57:49 then on in interface ethic where dhcp and dns up again edit icon and click there 22:57:58 you can see route ip there? 22:58:32 can u ping any of other services ip? 22:58:41 apart from bootstrap? 22:58:57 default gateway = 192.168.10.1 which is correct 22:59:15 you mean ping the containers? 22:59:30 yeah 23:00:15 like 192.168.10.110 23:00:16 like 192.168.10.111 23:00:25 they are working 23:01:03 bryan_att, i think looks like way bridge is configure have issue when system reboots 23:01:41 but in your case there should not be any configs right 23:02:12 I didn't configure the controller or compute nodes with any specific network info 23:02:22 Maybe the jumphost config is wrong? 23:02:43 https://www.irccloud.com/pastebin/Cx6Lunbd/ 23:03:46 can you run route on it? 23:03:52 route command 23:04:23 also ssh -v ubuntu@192.168.10.21 will give more information? 23:04:53 or ping -v 192.168.10.21 and ping -v 192.168.10.110 23:05:00 lets see the difference 23:05:05 https://www.irccloud.com/pastebin/dImExWnr/ 23:05:35 https://www.irccloud.com/pastebin/o0uzuxSb/ 23:06:16 https://www.irccloud.com/pastebin/AZuSObZM/ 23:09:37 should I have network-manager disabled on the jumphost? It says the wired connection is "unmanaged" which wasn't the same before I booted. 23:10:13 https://www.irccloud.com/pastebin/IJKiQO8i/ 23:16:05 also can you log in into juju ssh juju-gui/0 23:16:07 and send me cat /etc/network/interfaces file output? 23:16:12 bryan_att, looks like my ubuntu charm is rewriting the interfaces file which is not valid 23:16:14 so if comes up initially when but it reboots it switches to dhcp while server is in states mode 23:16:15 i am finding this on my bootstrap node auto juju-br0 23:16:15 iface juju-br0 inet static 23:16:15 bridge_ports eth0 23:16:15 gateway 10.4.0.1 23:16:15 address 10.4.2.20/21 23:16:15 mtu 1500 23:16:36 if thats the case then i need to restate my charms and interfaces.host file need to be corrected 23:16:46 earlier it used to be dhcp rather than static 23:16:56 ok, let me get the files for you 23:18:19 https://www.irccloud.com/pastebin/BQ5VcAic/ 23:20:22 bryan_att, i think this is issue with charm i wrote may be you can have a look into maas log file and grep for ip 23:20:29 if it's DHCP it would still need to assign the same IPs to the MAC, right? Because the service addresses should not change just cause the system is restored 23:20:35 for mac in /var/log/syslog file 23:20:45 you need to login into charm 23:20:52 sorry masa 23:20:56 maas 23:21:02 ok 23:21:04 then search for mac in /var/log/syslog 23:22:04 search for the word "mac"? 23:24:12 https://www.irccloud.com/pastebin/Vi5UeW3N/ 23:24:40 these are lines for the mac of the controller (:45) and compute (:ed) 23:25:12 network 192.168.10.1 23:25:19 i am doubting this section 23:25:46 it should be network 192.168.10.0 but how does it matter 23:26:27 yes, but I can ssh to those addresses and they are the controller and compute nodes! 23:27:06 so for some reason MAAS has the old IPs and new ones were assigned by DHCP... 23:29:33 yes, the "network 192.168.10.1" line is strange. let me check my install guide... 23:30:10 sure engough, that's what I wrote on the wiki: https://wiki.opnfv.org/copper/academy/joid 23:30:22 Strange that it worked before. 23:32:16 I can change it and restart networking, but the DHCP allocation for the controller and compute is clearly not expected, right? at least it should provide the MAAS-known addresses (.21, .22, and .23) 23:32:52 i think i have change in interfaces.host 23:33:02 i am working on fix and let you know soon 23:33:14 ok, thanks - brb 03:23:41 Hi,all. when i excute the 02-maasdeploy.sh,there is something wrong,i don't know why? 03:24:14 2015-12-09 19:44:17,536 DEBUG Command executed successfully: stdout='Import of boot resources started' 03:24:16 2015-12-09 19:44:17,536 DEBUG Logging into 192.168.122.2 03:24:17 Importing images ... Queued for download 03:24:19 Importing images ... Downloading 5% ad 03:24:20 Importing images ... Downloading 5% 03:24:22 Importing images ... Downloading 5% 03:24:23 Traceback (most recent call last): 97% 03:24:25 File "/usr/bin/maas-deployer", line 9, in 03:24:26 load_entry_point('maas-deployer==0.0.1', 'console_scripts', 'maas-deployer')() 03:24:28 File "/usr/lib/python2.7/dist-packages/maas_deployer/cli.py", line 88, in main 03:24:29 engine.deploy(target) 03:24:31 File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/engine.py", line 71, in deploy 03:24:32 self.wait_for_import_boot_images(client, maas_config) 03:24:34 File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/engine.py", line 330, in wait_for_import_boot_images 03:24:35 complete, status = checker.are_images_complete() 03:24:37 File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/maasclient/bootimages.py", line 93, in are_images_complete 03:24:38 status = self.get_status() 03:24:40 File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/maasclient/bootimages.py", line 60, in get_status 03:24:41 {'host': self.host, 'sequence': self.sequence.next()}) 03:24:43 StopIteration 14:33:28 narindergupta: hi 14:33:42 David_Orange, hi 14:33:59 how are you ? 14:34:14 David_Orange, i am fine and trying to work out a problem 14:34:38 David with nodes interfaces. But yeah do you know how was installation? 14:34:52 ok, about the bundle yaml file, osd-reformat: 'yes' value 14:35:18 yeah 14:35:35 was there any issue? 14:35:37 is it possible to set it to yes or no, and not use comment 14:35:47 it just about parsing for ci 14:36:09 sure no problem we can have it no as default 14:36:15 and move to yes if needed 14:36:19 ok, good for me 14:36:33 or let me check default first 14:36:42 sure 14:38:05 by default i no 14:38:56 i will make the change accordingly apart from that everything was good in orange pod correct? 14:39:11 we know known ODL issue related to groups 14:40:22 i am installing it with manual ci procedure 14:40:34 yes i see 14:41:11 you need more help on to debug this version or make the latest one working 14:41:12 ? 14:42:06 yeah 14:45:22 My colleague Bertrand is installing a new pod for testing purpose. When available we could use it to test new odl version with my colleague working on odl 14:46:33 but we just received the servers, so i am not sure we it will be available before christmas holidays 15:06:57 narindergupta: i got an issue on orange pod: http://pastebin.geany.org/qPE8L/ do you have an idea ? 15:18:24 David_Orange, this is something new which i never faced before just wondering can we produce it on same setup? 15:19:35 i clean first ? 15:27:55 yes please? 15:28:09 delete the files from ~/.juju/* 15:28:17 rm -rf ~/.juju/* 15:29:30 done, deploy is ongoing 15:30:05 in parallel i try to connect to intel pod5 15:31:02 vpn is up, but i cannot find the jumphost ip (i see node list in https://wiki.opnfv.org/get_started/pod_5_-_specs) but nothing about account or jumphost ip 15:31:18 did i miss something ? 15:39:52 narindergupta: the deploy has pass the last error, don't know what happened 15:40:10 David_Orange, ok thanks 15:41:17 node1 is jump host 10.2.65.2 15:41:44 which user ? 16:32:56 fdegir, hi good morning 16:33:12 David_Orange, ubuntu is default user 16:33:24 but i think fdegir is working on same as well 16:35:17 hi 16:35:24 narindergupta: ok, i have a deny on publickey reason. I will see that with fdegir 16:35:27 pod5 is a ci pod 16:35:30 hi fdegir 16:35:55 and the job for it has already been created 16:36:22 hi David_Orange 16:36:50 fdegir: narindergupta: ok, so where can i prepare then test joid ci install ? 16:37:15 David_Orange, do you want to use pod6? 16:37:39 narindergupta: if it is available 16:37:53 David_Orange, for time being you can use it 16:38:13 i am working on few fixes but that should be fine i guess for now 16:38:30 narindergupta: ok, i check connection to it 16:38:42 sure 16:39:27 fdegir: i worked on joid ci, do you prefer i abandon the actual push or do you prefer another patch ? I rewrote scripts and jobs based on your comments 16:44:09 narindergupta: ssh ubuntu@10.2.66.2 with my key failed with a permission denied, can you check my key is present ? it ended with david.blaisonneau@orange.com 16:44:34 i think i have add a key 16:44:54 can u point me to your public keys? 16:45:47 got it no worries 16:46:01 can u try now 16:46:03 ? 16:46:35 great 16:46:38 thanks 16:48:54 cool 16:49:22 fdegir, i think i have fixed the issue of permission denied can u try now to rerun your job? 16:49:50 bryan_att, i think have solution to your issue 16:50:45 bryan_att, let me know once you are ready to run? 16:58:30 David_Orange: please leave the pod5 for ci 16:58:37 David_Orange: you can continue working jobs for orange pod 16:58:45 sure, i will use pod6 16:58:54 David_Orange: the job I created for joid is the simplest possible to get joid work from jenkins 17:00:03 fdegir: ok, you already did one, can i see it ? 17:06:48 David_Orange: https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=blob;f=jjb/joid/joid.yml 17:07:36 narindergupta: I talked to the intel guys 17:07:46 narindergupta: they're going to add jumpserver for pod5 and pod6 17:07:54 narindergupta: so we have some work to do 17:08:02 narindergupta: you need to have 3 controllers + 2 computes 17:08:29 fdegir, ok sounds good to me. Thats a easy job in yams i have to change 1 to 2 17:08:33 narindergupta: and I need to reconnect the pod5 and pod6 jumphosts to jenkins 17:08:34 narindergupta: not the actual nodes 17:08:37 and add new nodes mac into deployment.yaml 17:08:42 narindergupta: good 17:08:57 not the actual nodes means? 17:09:20 fdegir, we can make new server as compute node 17:09:29 so no issues 17:09:46 narindergupta: the new server will be a bit older/different from the 5 servers you already have 17:09:55 just to serve as jumphost 17:10:27 ok no problem i will reset it up once I have more info 17:10:51 fdegir, meanwhile we can use the existing setup 17:10:58 yes 17:11:05 until they fix the stuff 17:11:07 we go this way 17:12:21 cool can u try to rerun the test today? 17:14:18 narindergupta: shortly 17:14:32 narindergupta: did some updates to the jenkins pod configuration 17:14:35 in 5 minutes 17:15:00 fdegir, ok 17:17:30 narindergupta: https://build.opnfv.org/ci/job/joid-deploy-intel-pod5-daily-master/1/console 17:19:00 fdegir, thanks looks like its running 17:19:01 David_Orange: please refresh joid.yml file 17:19:12 there was a small issue which's been fixed now 17:20:13 you do a maasdeploy without deleting MAAS VM and bootstrap ? 17:20:50 David_Orange: that's what I'll do later on 17:20:58 narindergupta said this should work as well 17:21:08 David_Orange, maas-deplyer do it automatically for you 17:21:20 that is what i am doing in destroy-vm.sh 17:21:29 yes, forgotten stuff 17:21:59 David_Orange: we can easily replicate this job for orange pod 17:22:18 and point to a different config file for orange pod 17:22:57 narindergupta: ok, so i dont need to add delete vm 17:24:22 David_Orange, both way it will work in mass-deployer there is force which i am using current;y 17:25:05 which will delete the existing vm and create new one if already present if not it will create 17:25:59 narindergupta: fdegir: is it necessary that i continue to work on my patches ? i really have the feeling to be useless on the ci task 17:27:10 David_Orange: nothing is useless 17:27:26 David_Orange: I think it is better if someone takes the responsibility of maintaining this 17:27:34 David_Orange: and you have the joid experience 17:27:44 David_Orange: once we have the basics, you can evolve it 17:27:57 the ci part I mean 17:28:48 David_Orange: you also started doing stuff while things are still in the air 17:28:58 David_Orange: there is a great push and changes in stuff to make the release 17:29:06 yes, i learn on jenkins, for me it is usefull, but you are much more efficient than me 17:29:26 David_Orange: please don't stop what you started 17:29:40 David_Orange: we always need people like you to help us with the stuff 17:30:02 fdegir: dont worry, it is not a problem, but i will let you finish something working then follow your steps 17:30:18 David_Orange: I think this is good enough for joid on pod5 17:30:31 I'll continue building on it and add functest, yardstick once we get the jumphost 17:30:42 so you can fix orange pod in parallel to this 17:30:49 and add more while you move one 17:30:52 on 17:34:07 ok, i will let you finish something working and adapt it to orange pod. 17:35:20 ok 17:35:54 end of day for me, bye and have a nice day 17:41:21 bye 17:41:26 narindergupta: I'll also leave 17:41:42 the deployment is running 17:41:49 will see how it ends 17:41:55 how much time it takes normally? 17:49:43 40 - 45 minutes 19:51:39 fdegir, looks like my deployment got completed but it marked as failed do you know why it was marked as failed? 20:12:36 bryan_att, please review this for your fix https://gerrit.opnfv.org/gerrit/#/c/4227/ 20:12:52 where reboot of the server you could not login into the server? 20:49:44 bryan_att, i saw your comment basically it will not use our interfaces file charm for OS installations. So that static ips configured by MAAS will be kept as soon as we follow the process of bringing up the machine after shutdown. You need to deploy it again to test it. So test case is deploy the openstack using ./deploy.sh and then reboot any of the server. 20:50:55 soon we will get rid of this when MAAS is fully functional with 1.0 I just logged a bug to MAAS though 20:51:51 where commissioning does not determine the other two interfaces on nodes. 06:35:53 Hi,all. I have excute the 02_maasdeploy.sh successfully using default lab. But I saw the bootstrap.maas fail commsioning? why? 10:05:07 narindergupta: jenkins fails the job if any of the commands/scripts executed exits with non-zero 10:05:56 narindergupta: deploy.sh seems to be failed execution 10:06:12 fdegir, as i can see deployment was completed so need to check why 10:06:45 grep command exits non-zero 10:06:47 probably 10:07:37 grep should exit with non zero but after that i am running few other commands 10:08:04 i am hoping you are not checking the grep status 10:10:13 https://gerrit.opnfv.org/gerrit/gitweb?p=joid.git;a=blob;f=ci/deploy.sh 10:10:44 you can see i am running echo command and then creating the opener file 10:11:12 check_status checks the status of the deployment 10:13:05 i am running a manual deployment in pod6 lets see how it reacts? 10:18:36 I don't 10:18:38 but jenkins does 10:18:54 so you have to handle grep exit status in your script 10:21:47 yes it was part of function and upon exit of function i am running other commands 10:34:17 hi 10:35:48 David_Orange, hi 10:35:54 David_Orange, how are you? 10:36:06 fdegir: do you have 5 min ? about how to work on your joid.yaml without disturbing you 10:36:21 David_Orange: sure 10:36:29 narindergupta: fine and you ? you never sleep ? 10:36:48 David_Orange, actually i slept early so got up early 10:36:59 though of testing few things 10:37:52 narindergupta: thanks to you, not always easy to have an go sleep on internatiol project 10:38:17 yeah i can understand 10:39:10 fdegir: how can i do, I take your file, add orangepod and add another jobs ? 10:40:57 fdegir: i merge some jobs, and scripts, some there are less of them, but i have one last question. When we set scm, the workspace is created with the content of the git at the startup, removing everything in the previous folder, ture ? 10:40:59 David_Orange: we're in the process of moving pod config to a separate file 10:41:06 David_Orange: so please first update this file 10:41:07 https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=blob;f=jjb/opnfv/pod-params.yml 10:41:18 David_Orange: and add orange slave into it 10:41:30 ok 10:41:33 David_Orange: about git stuff 10:41:45 David_Orange: we have by default wipe out the workspace so it is always clean start 10:41:57 David_Orange: you can safely use $WORKSPACE 10:42:18 David_Orange: $WORKSPACE is an env var injected to all jobs by Jenkins by default 10:42:32 David_Orange: and it always points to / 10:42:35 ok 10:42:45 David_Orange: once you update the pod-config 10:42:57 David_Orange: then you can work on joid.yml 10:43:02 understood 10:43:10 https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=blob;f=jjb/joid/joid.yml 10:43:30 David_Orange: you have this pod var in joid.yml on line 12 10:43:40 David_Orange: you need to convert that to list 10:43:46 ok, i set pod-params, abandon my review and work on your base 10:44:07 David_Orange: and add orange pod there, matching the pod name you've given in pod-params.yml 10:44:09 sure 10:44:14 so 10:44:21 if you look at job names in joid.yml 10:44:28 they have {pod} in them 10:44:32 yes 10:44:35 when Jenkins Job Builder sees that 10:44:44 and sees list of pods above 10:44:51 it creates 1 job per pod 10:45:09 same is valid for stream etc that is in job name 10:45:22 this is the first part 10:45:32 I suppose you will need to do things differently for orange pod 10:45:36 ok, lets say i add a 'joid-custom' for jenkins install out of ci, how can i do ? 10:45:57 is joid-custom a separate job? 10:47:00 isn't it the best way to avoid conflict between your work on pod5 and mine on orangepod2 ? 10:47:21 we can do that by using same jobs 10:47:36 by creating separate builders based on pod 10:47:57 builders are the things that do stuff like executing shell scripts and so on 10:48:05 so I can have a separate builder for pod5 10:48:12 and you can have another one for orange 10:48:32 you update orange builder without touching intel one 10:48:46 fdegir, i am able to reproduce issue and fixing it. 10:48:48 but the jobs do the same thing; executing whatever builder that needs to be execute 10:49:09 let me convert my builder part to a builder macro 10:49:17 so you can create a macro for orange 10:49:58 ok, good to me, it will easier me understanding 10:53:59 David_Orange: https://gerrit.opnfv.org/gerrit/#/c/4329/ 10:55:54 David_Orange: you should be able to have multiple shells in same builder 10:56:34 fdegir: +1, ok, i understand 10:58:46 David_Orange: sent a new patch and separated shell script executions so narindergupta can see which script gets executed 10:58:47 https://gerrit.opnfv.org/gerrit/#/c/4329/ 10:59:02 David_Orange: I'm going for lunch and will be in meetings during the afternoon 10:59:11 David_Orange: will check gerrit time to time 10:59:34 i have to go too, i post orangepod to config 10:59:42 fdegir: thanks 10:59:50 ok 10:59:52 np 11:00:33 fdegir: just one last thing, you prefer script in the yaml or outside ? 12:11:33 fdegir, can u run job again in intel pod5? I have the code checked in with correct status 12:13:10 narindergupta: https://build.opnfv.org/ci/job/joid-deploy-intel-pod5-daily-master/2/console 12:13:19 thanks 12:13:32 David_Orange: if the script is huge, you can create a separate script and use include 12:14:23 David_Orange: if couple of lines, you can perhaps leave it in yaml 12:48:08 narindergupta: syntax error 12:48:32 fdegir, ok will look into it 12:56:21 fdegir, submitted code again please retry? 12:57:38 narindergupta: https://build.opnfv.org/ci/job/joid-deploy-intel-pod5-daily-master/3/console 13:32:34 fdegir, i think clean up script was not proper. Will you please rerun the test? 13:33:25 narindergupta: rerunning in same way you mean? 13:33:35 including cleanup script 13:34:20 narindergupta: https://build.opnfv.org/ci/job/joid-deploy-intel-pod5-daily-master/4/console 13:35:42 looks like its hanging again 13:35:56 can u kill this job and rerun it? 13:38:57 narindergupta: https://build.opnfv.org/ci/job/joid-deploy-intel-pod5-daily-master/5/console 13:39:10 failed 13:39:29 ok let me have a look 13:42:31 fdegir, can retry now? 13:43:43 narindergupta: https://build.opnfv.org/ci/job/joid-deploy-intel-pod5-daily-master/7/console 15:34:50 fdegir: ping 16:03:41 fdegir: how idd the test do? 16:04:06 fdegir: i stopped monitoring for a while? 16:12:54 narindergupta: me too 16:13:01 please check the log from the link 16:13:20 fdegir: will you please send me link again as i changes my OS? 16:16:45 fdegir: sorry for my double review, working with 2 pc is not good for clean review 16:18:02 fdegir: the new one passed jenkins 16:24:45 David_Orange: no problem 16:24:47 fdegir: got one more error but this time it should pass. Plesae re run it again? 16:25:26 narindergupta: https://build.opnfv.org/ci/job/joid-deploy-intel-pod5-daily-master/8/console 16:25:37 narindergupta: sorry for asking but don't you test these things before committing? 16:30:47 yes i do but i modify it locally and while converting to the system gerrit having issue 16:35:06 narindergupta: we need virtual deployment jobs for joid to catch these issues there 16:36:16 fdegir: you meant single node install? or hide the deployment? 16:36:32 virtual with kvm 16:43:33 fdegir: yeah we can do that provided you have kvm created 16:43:53 narindergupta: rest of the installers have it 16:44:07 narindergupta: we only provide the server with virtualization capabilities 16:44:15 narindergupta: the rest is up to you, creating vms, etc. 16:44:34 oh ok no problem i can build it by default option 16:44:47 narindergupta: apart from ci stuff 16:45:02 for example if no lab info is passed then we can use the same installer 16:45:09 narindergupta: it is good for users that want to try joid out and don't have data center at their homes 16:45:23 just saying 16:46:12 fdegir: yeah understand for them we already have another tool called openstack-installer which will created the openstack experience on VMs in a container 16:46:54 may be virtual environment i can include that tool as well will be helpful. we are README in joid alreday mention that but not integreted into CI 18:01:30 fdegir: hurray this time it passed. + chmod 0600 cloud/admin-openrc + echo '...... finished .......' ...... finished ....... Finished: SUCCESS 22:01:16 narindergupta: happy to hear! 22:01:26 narindergupta: I'll set nightly triggers 22:01:53 fdegir: thanks and i have a patch for virtual install as well 22:02:09 i have submitted for review already