17:00:10 #startmeeting Joid weekly 17:00:10 Meeting started Wed Dec 9 17:00:10 2015 UTC. The chair is arturt. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:10 Useful Commands: #action #agreed #help #info #idea #link #topic. 17:00:10 The meeting name has been set to 'joid_weekly' 17:00:54 #info David Duffey 17:01:24 #info Artur Tyloch 17:01:26 Hi arturt 17:01:42 narindergupta: have you started gotomeeting ? 17:02:14 when I am trying to join gotomeeting it shows message meeting in progress 17:02:58 arturt, waiting for you 17:04:14 #info David Blaisonneau 17:04:21 #topic agend bashing 17:04:30 #link https://etherpad.opnfv.org/p/joid 17:07:16 #info Iben Rodriguez 17:08:00 #topic sprint backlog review - MAAS Pilot 17:09:55 #info #link MaaS Pilot location list https://wiki.opnfv.org/pharos/maas_getting_started_guide#maas_pilot 17:15:13 #info * Orange Pod 2 - https://wiki.opnfv.org/opnfv-orange - working to resolve a configuration issue and reinstall 17:15:38 #info * Intel Pod 5,6 https://wiki.opnfv.org/get_started/intel_hosting - eta dec 15 DavidB@Orange 17:15:39 17:17:57 #info * Juniper Pod 1 - Iben working with Samantha@canonical to document the setup 17:19:38 #info 17:19:38 * Dell Pod 2 - https://wiki.opnfv.org/dell_hosting - TPM Mark Wenning @canonical and Vikram @Dell 17:21:34 #info * UTSA - TBD 2016 17:21:42 #info * CENGN - TBD - Nicolas@Canonical 17:21:54 #info * Huawei - TBD Prakash to inform us on status 17:23:19 #topic sprint backlog review - JOID-25 17:23:28 #topic sprint backlog review - JOID-29 17:30:04 #topic sprint backlog review - JOID-25 17:41:15 #info #link JOID charm (plugin) more info : https://wiki.opnfv.org/joid?&#juju_and_maas_training_materials 17:41:15 [08:20am] <— sent to pharos project meeting this morning 17:42:45 #info please review the #link for the pharos meeting minutes http://ircbot.wl.linuxfoundation.org/meetings/opnfv-pharos/2015/opnfv-pharos.2015-12-09-16.00.html for more info on writing charm 17:44:24 https://jujucharms.com/docs/stable/authors-charm-writing 17:46:30 #info added #link Your first charm starts here! to wiki 17:46:53 : #info added #link Your first charm starts here! to wiki https://jujucharms.com/docs/stable/authors-charm-writing 17:47:34 #topic sprint backlog review - JOID-16 17:56:25 #action iben update https://wiki.opnfv.org/joid/get_started with info about Jenkins installation in POD 17:56:40 #info #link https://wiki.opnfv.org/octopus/jenkins_slave_connection how to connect jenkins slave 17:56:56 #action David_Orange update https://wiki.opnfv.org/joid/get_started with info about JOID CI job for a new POD 17:57:57 #topic sprint backlog review - JOID-25 17:58:16 * fdegir connected pod5 to opnfv jenkins 17:59:04 about jenkins installation; please do not install jenkins to any PODs that is going to be used by OPNFV CI 17:59:28 #info https://wiki.opnfv.org/joid/get_started <— updated - please review and advise how to improve 17:59:33 hello fdegir 17:59:40 hi 17:59:46 i will let them know on audio 18:00:01 thanks 18:00:03 I already explained this to narindergupta 18:00:07 so he should be aware of this 18:00:11 k 18:01:10 but it seems pod5 lacks jumphost 18:01:15 that's a problem 18:04:00 fdegir, pod5 has jumphost 18:04:19 narindergupta: the ip I got from you is the ip of the node1 18:04:25 narindergupta: and pod5 has 5 nodes 18:04:33 yeah that is jumphost 18:04:41 as intel has only five servers 18:04:43 narindergupta: but pharos spec says 5 + 1 18:04:59 narindergupta: and all the test projects rely on the jumphost 18:05:04 yeah both intel pods 5 and 6 hs total 5 18:05:16 narindergupta: I noticed that as well 18:05:24 and my deployment uses 3 control and 1 compute 18:05:43 narindergupta: but some test projects probably expect 2 computes 18:05:52 narindergupta: yardstick for example 18:06:59 narindergupta: you have the mail from Ana 18:07:18 narindergupta: and she says: "We need a baremetal system, 3 x controllers, 2x compute, 1x jump host." 18:08:11 fdegir, ok it is something intel has to provide. 18:08:26 narindergupta: I reported this to them already 18:08:41 fdegir, ok thanks 18:16:27 narindergupta: until this is resolved 18:16:27 narindergupta: we can start creating jenkins jobs for joid 18:17:08 fdegir, David from Orange is helping us on that. And i believe he is working with you on the same. 18:17:25 narindergupta: what he does it for testing purposes 18:17:48 narindergupta: and he uses stuff (scripts, etc.) that is locally available on their pod 18:17:56 narindergupta: which the CI doesn't allow 18:18:07 narindergupta: everything must come from your OPNFV git repo 18:18:15 narindergupta: deployment, cleanup, and so on 18:18:38 narindergupta: so David's job is not a valid job for CI 18:24:16 fdegir, everything we have in joid git i think once we have valid script 18:24:27 then we are planning to include those in repo 18:24:40 all scripts are part of ci and git 18:42:05 narindergupta: so it is not possible to create CI stuff as of yet 18:44:26 fdegir, problem is i do not know what to do for CI and we asked help from David. If some one can lead me with examples then i think of how to automate? 18:46:08 narindergupta: all the other installers do this already 18:46:13 narindergupta: they're the examples 18:46:43 narindergupta: you should have a ci directory in your repo and place the scripts that are supposed to be executed by CI in this directory 18:46:53 will you point me exact scripts which does that? And it should be on jump host or somewhere else 18:47:08 fdegir, i have the same way 18:47:16 under joid i have ci directory 18:47:24 which has script to do deployment 18:47:31 like deploy.sh and clean.sh 18:47:35 narindergupta: then what did you mean with "we are planning to include those in repo" 18:47:44 narindergupta: and "once we have valid script" 18:48:16 fdegir, yeah David is working on scripts based on joid and once we have those we can include in joid thats what i meant 18:48:27 narindergupta: so they're not there yet 18:48:56 narindergupta: well, I see big risk here 18:48:58 fdegir, yes for that once David done the completion i have plan to include those specific ones 18:49:05 narindergupta: we can't even deploy stuff from jenkins 18:49:12 narindergupta: test projects are in the pipe 18:49:20 narindergupta: and we don't even have the jumphost 18:49:39 fdegir, i am more confused now. 18:49:52 fdegir, earlier you mentioned that we do not need jenkins 18:50:29 I said you do not need Jenkins installation on your pod 18:50:57 fdegir, so what i need to ci then? 18:51:10 you need to have automatic deployment scripts for CI 18:51:22 once you have those scripts, you also need jenkins jobs to execute those scripts 18:51:37 on jumphost 18:51:41 fdegir, we have scripts already to do deployment 18:51:53 david is working on jenkins scripts 18:51:59 then what are the scripts you are waiting for David to fix? 18:51:59 and document the same 18:52:08 jenkins script 18:52:59 git clone http://gerrit.opnfv.org/gerrit/joid.git is our joid code 18:53:23 but David is using everything from local copy 18:53:35 so noone can guarantee it will work 18:53:49 for CI 18:54:18 it can work on Orange POD - that's OK if they want to go that way 18:54:50 but it won't work in CI since all the paths in the scripts created by David points to the home directory of whatever user 18:59:53 basically we are integrating th ejoid to labs which are participating and putting config files in each lab directory and as part of joid itself 19:00:33 so you can git clone the joid and then run it from joid/ci and deploy.sh copies to joid/ci and run deployment everything from there 19:01:41 when I look at David's patch, I see lots of things there 19:02:17 virsh commands for cleanup, recovery, config, etc 19:02:30 they are provided by projects normally 19:02:33 not by ci 19:05:57 yes those are parts of our reply.sh file 19:06:16 sorry deploy.sh to create a maas and bootstrap vm and automated 19:07:49 I don't know, I'm pretty confused right now 19:07:56 what comes from where, who does what 19:09:13 can u give me one example ci command ci run after clone the git for installer 19:13:21 I can give you two 19:13:25 https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=blob;f=jjb/compass4nfv/compass4nfv.yml 19:13:43 lines 475 to 505 19:14:10 https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=blob;f=jjb/fuel/fuel.yml 19:14:12 line 321 19:14:16 which executes 19:15:04 https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=blob;f=jjb/fuel/fuel-deploy.sh 19:15:14 and apex is doing this as well 19:20:28 fdegir, looks like it should be easy as in case of joid ./deploy.sh --dha $CONFDIR/dha.yml --network $CONFDIR/network.yml it should be 19:20:29 narindergupta: still looking at David's work 19:20:34 narindergupta: he has pod config 19:20:46 for orange pod I suppose 19:20:48 do you have pod config for intel pod5? 19:20:59 fdegir, ./deploy.sh -o liberty -s odl -t ha -l intelpod5 19:21:06 yes 19:21:12 where is the config located? 19:21:34 everything under maas/intel/pod5/deployments.yaml 19:22:00 good! 19:22:05 so Maas installer run 02-maasdeploy.sh intelpod5 19:22:18 so i think basic work is already there 19:22:33 please don't misunderstand me btw 19:22:39 just trying to understand where you are 19:22:43 not at all 19:22:46 and how can I help you to come to CI 19:22:50 anyway 19:22:57 as i just want to know if there is risk or gap how to resolve it 19:22:58 this command will do the deployment 19:23:08 ./deploy.sh -o liberty -s odl -t ha -l intelpod5 19:23:19 any pre or post steps like David does? 19:23:26 do we need cleanup? 19:23:32 before issuing above command? 19:23:51 i believe clean up is done before 19:24:11 ok there are two cases here i think 19:24:34 first is fresh install with first time install. 19:25:02 then run ./02-maas-deployment.sh intelpod5 this is create maas and bootstrap vm 19:25:29 then run ./deploy.sh -o liberty -s odl -t ha -l intelpod5 will do the deployment 19:25:46 if its already deployed once then ./clean.sh and ./deploy.sh -o liberty -s odl -t ha -l intelpod5 will deploy it 19:26:04 I think other installers do fresh install 19:26:48 if need fresh installs then run ./02-maas-install.sh intelpod5 again and then ./deploy.sh -o liberty -s odl -t ha -l intelpod5 19:27:20 i have method for both 19:28:19 installer does not have to re created again and again 19:28:53 and our ./clean.sh will clean all the environment except installer vm 19:29:45 ok 19:29:56 so if one wants to clean the installer vm as well? 19:30:29 which is what is done by other installers if I'm not totally mistaken 19:31:07 that's why David has that joid-destroy script then 19:31:17 so the process on CI will be 19:31:21 -destroy-vm 19:31:28 -02-maas-install 19:31:31 -deploy.sh 19:32:03 no need to destroy VM 19:32:11 mass-install will destroy iy 19:32:12 it 19:32:22 just run 02-maasdeploy.sh 19:32:24 but what if we do in order to make the ci processes same for all the installers 19:32:44 sure no issue then run ./clean.sh 19:32:46 we always do fresh install - including creation of VMs 19:32:49 destroy the vm 19:33:02 and then run 02-maas-deploy,sh 19:33:07 then deploy.sh 19:33:12 ok 19:33:30 I'll do this quickly, probably tomorrow 19:33:30 and come back to you 19:33:37 do you use pod5 for any reason? 19:33:45 it'll become ci pod meaning hands off 19:34:03 sure no problem. I think David is working on modifying other within config as well. 19:34:17 ok 19:34:21 like ceph-monitor password etc.. 19:34:34 so that same won't be used for security purpose 19:34:43 other? 19:34:48 you mean orange pod? 19:35:37 anyway, thanks for the info, this was really helpful 19:35:50 will send you the logs when I have them 19:42:06 fdegir, i think we doing for orange which can excerize on all labs 19:42:37 other configuration parameter within openstack 19:42:53 like passwords and any other secret information. 19:46:00 narindergupta: we have a new repo which is not readable to the world 19:46:13 we plan to move this type of sensitive information to that repo 19:46:27 ok 19:46:28 but before doing this 19:46:43 do I need all this secret stuff fixed for intel pod5 19:46:48 or will it just work? 19:46:48 in that case it needs to segregate from JOID pretty soon? 19:47:00 no not at all we have default values 19:47:06 ok, good 19:47:07 it will just work 19:47:12 about the secure repo 19:47:25 you're part of the discussion 19:47:25 Frank, Jonas, Tim, Weidong 19:47:25 yeah 19:47:32 once something comes out of that discussion 19:47:39 that something should end up in this new repo 19:47:40 about the config file you meant? 19:47:43 yes 19:47:52 all this type of password, ip, mac, etc. etc. 19:48:10 deployment config, pod config, and so on 19:48:20 yeah we will taking those and replacing in installer defined repo 19:48:39 yep 19:48:43 got you i see where we are going and make sense 19:48:58 until that happens, we have to use what we have 19:49:02 and switch to them once they're settled 19:49:29 I have to drop now 19:49:33 yeah no problem as once settle then installer needs to write parse engine and modify accordingly 19:49:38 sure please 19:49:44 talk to you tomorrow 20:08:28 narindergupta: https://gerrit.opnfv.org/gerrit/#/c/4161/ 20:11:51 fdegir, looks like you had the template. and modified the one and everything looks good to me. Were yo able to run an dverufy 20:12:40 narindergupta: will run it shortly 20:12:44 once a releng committer gives +2 20:12:52 ok 20:13:20 joid there is a change 20:14:10 02-maasdeploy.sh intelpod5 is correct command 20:14:19 ./02-maasdeploy.sh intelpod5 20:15:33 narindergupta: thanks, fixed that 20:15:38 please have a look at that part again 20:17:30 yes looks good now lets verify it if issue exist then i am happy to assist 20:18:01 may i know one moe thing this builders: tag is for deployment right 20:18:13 as joid do not have any build script to follow 20:19:07 our charms do it run time if needed 20:20:19 tag? 20:23:37 there is option builders: under which we have deploy shell commands 20:27:46 yes 20:27:49 you don't have to have build 20:27:55 it is jenkins term 20:27:59 it calls everything as build 20:28:10 ok 20:40:02 narindergupta: it failed 20:40:25 narindergupta: https://build.opnfv.org/ci/view/joid/job/joid-daily-deploy-master/1/consoleFull 20:43:12 fdegir, yeah this portion i added today but could not verify 20:43:45 will work on it basically this will create cloud/openrc file which contains the file to login into openstack 20:44:33 ok 20:44:50 let me know when new version of the script is in the repo so I can give it a try 20:47:54 sure 20:48:03 fdegir, also do you know why opnfvsdn=''\''odl'\''' 20:48:15 is that way i thought it should be clean old only 20:48:18 old only 20:48:44 are you talking about deploy script or? 20:49:42 ok 20:49:43 got it 20:49:47 it was a problem in my part 20:49:51 and fixed that already 20:50:07 narindergupta: check this log please 20:50:08 https://build.opnfv.org/ci/view/joid/job/joid-daily-deploy-master/2/consoleFull 20:50:26 quotes are not there anyomre 20:50:27 opnfvsdn=odl 21:21:05 fdegir, thanks and i have checked in the fixes 21:21:12 will you please retry it now? 21:22:29 fdegir, also i am not seeing output of ./02-maasdeply.sh file? Was that ran successfully? 21:22:31 narindergupta: failed with different error 21:22:31 https://build.opnfv.org/ci/view/joid/job/joid-daily-deploy-master/3/console 21:23:13 narindergupta: could this be the reason? 21:23:15 sudo: no tty present and no askpass program specified 21:23:25 jenkins user might need passwordless sudo 21:24:02 yes jenmkins needs passwordless sudo 21:24:10 i can edit the profile 21:24:26 or you can 21:25:25 this one need maasdeploy script first 21:25:27 can you do that please? 21:29:05 yeah looking 21:36:59 fdegir, i fixed that but you need to run ./02-maasdeploy.sh intelpod5 as well for getting the API file 21:37:04 narindergupta: my power was out earlier, and I need to know how to bring the nodes and services back online - a good test. I have used the maas UI to power them controller and compute nodes back on, but I can't SSH to them and few of the services seem to be active per the status command 21:37:58 narindergupta: that script is there already 21:38:06 fdegir, ok 21:38:10 running it now? 21:38:18 bryan_att, sure 21:38:32 narindergupta: how long should I have to wait, and is there any manual action I need to take? 21:38:40 bryan_att, first you need to bring the jump host server 21:38:48 narindergupta: it seems this is better 21:38:49 * fdegir https://build.opnfv.org/ci/job/joid-daily-deploy-master/4/console 21:39:01 narindergupta: and for my manually created congress container, how do I start it again? 21:39:27 narindergupta: jumphost was powered on and maas/juju UIs are active 21:39:29 just check whether container is already started or not 21:39:44 use sudo lxc-ls 21:39:54 ok 21:40:16 returns nothing 21:40:29 (from the jumphost) 21:40:56 i think owe did install that on control nodes correct? 21:41:06 then juju ssh nodes-api/0 21:41:15 yes, on the control node 21:41:28 now run sudo lxc-ls 21:41:37 and check your container 21:41:49 https://www.irccloud.com/pastebin/dAqa0LAV/ 21:41:57 some error 21:42:31 I can 21:42:36 hun i am hoping you used the same user during install? 21:42:46 (can't) SSH to it 21:42:51 Yes, unbuntu 21:43:10 (ubuntu) 21:43:18 so can you shutdown down your control and compute node and bring up only control node 21:43:36 OK, via maas? 21:44:12 (or manually) 21:44:22 lets try manually 21:44:44 ok, shutting down by soft press on the power switch 21:45:07 ok thanks 21:46:22 i think you should shutdown the bootstrap node lso 21:46:24 also 21:46:52 as per google this happens if ip address gets changed 21:47:13 ok, I will need to connect to IRC on a different machine, hang on 21:47:22 ok 21:47:32 its like powering system on one by one 21:47:39 first jumphsot 21:47:57 make sure both maas and bootstrap nodes are up 21:48:16 then control node and then compute node 21:48:21 ok, powering jumphost back on 21:49:16 ok 21:49:58 how do I know when the maas and bootstrap nodes are ready, is it when the maas and juju UIs are working? 21:50:09 correct 21:52:28 do I need to poweron the bootstrap node via maas? That's what I did last time. 21:58:17 narindergupta: the juju tab is spinning ("connecting to the juju environment"). Seems I need to power the bootstrap node on by MAAS, right? 21:58:18 no bootstrap node is marked as auto power on in lvm itself 21:58:32 if not then yes 21:59:00 seems not, so I will start it via maas 21:59:07 ok yeah then 21:59:31 now try to power on control node first and wait 22:00:29 fdegir, good news maas deployment is completed 22:00:57 fdegir, now deploy.sh script should be running? 22:04:18 ok, JuJu UI is back. 22:08:17 narindergupta: what's next? 22:08:37 cool now startup control node 22:08:58 and run juju status --format=tabular 22:09:03 for observation 22:09:23 after say 2 3 minute you can start the compute node 22:09:41 and check juju status --format=tabular 22:10:19 narindergupta: deploy.sh failed 22:10:44 fdegir, whats the error? 22:10:45 and that's all from me for today 22:10:56 after 2-3 minutes from power on, or some other event in the status output? when I did this before (Arno) I had to wait 2-3 minutes after all OpenStack controller services were back, or the nova-compute would not be registered properly with the ccontroller, and I would have to restart it 22:11:10 narindergupta: https://build.opnfv.org/ci/job/joid-daily-deploy-master/4/console 22:12:06 fdegir, permission denied ok i will have a look 22:16:04 narindergupta: ok, I did "sudo juju status --format=tabular | grep lost" until only the compute node services were listed. I'll power it on now 22:17:04 sure thanks 22:24:24 narindergupta: so now I login to the controller and do the lxc-ls command? 22:24:40 yeah please? 22:25:02 juju ssh nodes-api/0 22:25:11 OK, still getting the error I did before 22:25:26 https://www.irccloud.com/pastebin/cLTmUSqC/ 22:25:53 does juju status --format=tabular looks good? 22:25:58 yes 22:26:21 ok lets try to get ip address of the system and we can try using ip address 22:26:32 ssh ubuntu@ 22:27:42 same error 22:27:44 https://www.irccloud.com/pastebin/CsUq42Bt/ 22:28:01 maybe I forgot the ubuntu 22:28:48 didn't seem to help 22:28:52 https://www.irccloud.com/pastebin/rKzBaWmd/ 22:29:11 do not use juju to ssh 22:29:17 just simple ssh 22:29:37 ok, that didn't work either 22:29:38 like ssh ubuntu@192.168.10.21 22:29:47 same error? 22:29:53 yes, as pasted 22:29:58 can we try compute?. 22:30:07 juju ssh nova-compute/0 22:30:15 Are you able to get to that host at all? I wonder if the first two issues at the following are relevant: http://edoceo.com/notabene/ssh-exchange-identification . 22:30:45 https://www.irccloud.com/pastebin/mzsabOvT/ 22:31:19 can u try without sudo? 22:31:54 https://www.irccloud.com/pastebin/sluGFH0i/ 22:32:42 should I not have using sudo for the juju commands? I thought it was required somehow. 22:33:23 bryan_att, not for juju and i am modifying the ownership in envuronments.yaml so that no sudo needed 22:33:58 OK, is there something I need to do to reset it, or maybe there 22:34:05 s some other issue? 22:36:30 can we run ssh -v 192.1.68.10.21 22:36:46 can we run ssh -v ubuntu192.1.68.10.21 22:36:50 can we run ssh -v ubuntu@192.1.68.10.21 22:37:28 I assume you mean ssh -v ubuntu@192.168.10.21 22:37:58 ok 22:38:20 https://www.irccloud.com/pastebin/syuyxCQl/ 22:40:50 seems something strange in the networking setup. Obviously the maas VM was able to boot the nodes via wake-on-lan. and the containers are working fine, as well as the native services (at least reported fine in the juju status command) 22:41:16 ubuntu@jumphost:~$ sudo juju status --format=tabular 22:41:16 [Services] 22:41:16 NAME STATUS EXPOSED CHARM 22:41:16 ceilometer active false local:trusty/ceilometer-44 22:41:16 ceilometer-agent false local:trusty/ceilometer-agent-20 22:41:16