16:05:03 <narindergupta> #startmeeting JOID 16:05:03 <collabot> Meeting started Wed Sep 7 16:05:03 2016 UTC. The chair is narindergupta. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:05:03 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic. 16:05:03 <collabot> The meeting name has been set to 'joid' 16:15:19 <narindergupta> #info https://www.irccloud.com/pastebin/y4ub3ohr/ 16:19:03 <narindergupta> #info include the directories and files will be removed in D release which we kept it for legacy purpose 17:53:33 <bryan_att> narindergupta: ping 17:53:50 <narindergupta> bryan_att, hi how are you? 17:54:31 <bryan_att> fine, just an issue with JOID deploy, cloned yesterday; tried twice and it ends with a keystone error 17:54:49 <bryan_att> https://www.irccloud.com/pastebin/dKStScXQ/ 17:55:01 <narindergupta> is it master build or stable ? 17:55:06 <bryan_att> Master 17:55:17 <bryan_att> Should I be using stable? 17:55:44 <narindergupta> it should not matter as both are same 17:55:56 <narindergupta> and i am hopinh juju status shows no error? 17:56:03 <bryan_att> JuJu status looks fine 17:56:15 <narindergupta> this is problem Unable to establish connection to http://api.None:5000/v2.0/tokens 17:56:22 <bryan_att> yes 17:56:50 <narindergupta> please send meyasml files 17:56:56 <narindergupta> bundles.yaml first? 17:57:00 <bryan_att> ok 17:57:08 <narindergupta> also your labconfig.yaml 17:57:19 <bryan_att> https://usercontent.irccloud-cdn.com/file/uCkPZydN/bundles.yaml 17:57:42 <bryan_att> https://usercontent.irccloud-cdn.com/file/LuHLdupX/labconfig.yaml 17:59:29 <narindergupta> ok in your bundles.yaml we have os-public-hostname: api.None 18:00:17 <narindergupta> bryan_att, please send me deployconfig.yaml as well 18:00:33 <bryan_att> https://usercontent.irccloud-cdn.com/file/SayJNE6E/deployconfig.yaml 18:09:54 <narindergupta> will you please run this command python genBundle.py -l deployconfig.yaml -s os-nosdn-nofeature-nonha > bundles.yaml 18:10:01 <narindergupta> and send me bundles.yaml 18:10:26 <narindergupta> it looks like some issue in your case while in my case almost similar value no issues 18:10:41 <bryan_att> https://usercontent.irccloud-cdn.com/file/73EATKP4/bundles.yaml 18:12:28 <narindergupta> bryan_att, in this file looks like everything is ok 18:12:39 <narindergupta> somehow your bundles.yaml is wrong 18:12:58 <bryan_att> is it generated upon the deploy? 18:13:02 <narindergupta> yes 18:13:13 <narindergupta> and placed under joid/ci 18:13:22 <bryan_att> what generates it? 18:13:45 <narindergupta> which option r u using in -f here? 18:14:04 <bryan_att> you mean in the deploy command? 18:14:10 <narindergupta> yes 18:14:21 <bryan_att> ./deploy.sh -o mitaka -s nosdn -t nonha -l attvirpod1 18:14:34 <bryan_att> not using -f 18:14:46 <narindergupta> that should not add api then 18:15:14 <narindergupta> will you please send me console log for deployment? 18:15:18 <bryan_att> sure 18:15:37 <narindergupta> it seems code is working and manually i verified it 18:15:51 <narindergupta> so we need to find out the error which is causing it 18:19:15 <bryan_att> https://usercontent.irccloud-cdn.com/file/F1ImkPIr/deploy.log 18:19:49 <bryan_att> I see "+ keystone_PUBENDPOINT=api.None" 18:20:26 <bryan_att> right after https://www.irccloud.com/pastebin/uCP3LUwt/ 18:21:26 <narindergupta> yeah here is the problem python genBundle.py -l deployconfig.yaml -s os-nosdn-none-nonha-xenial_mitaka_publicapi 18:21:33 <narindergupta> publicapi should not be added 18:23:19 <bryan_att> so I can patch it locally and retry? what do I change? or should I wait for a patch? 18:26:30 <narindergupta> bryan_att, do you have deployment.yaml in your local directory? 18:27:12 <bryan_att> in joid/ci ? no 18:27:42 <narindergupta> bryan_att, ok that was root cause of issue. I will work on it meanwhile please copy that from ~/.juju/ 18:27:59 <narindergupta> and cleanup and restart the deployment 18:28:36 <bryan_att> ok, maybe this was an issue with how I redployed without redeploying maas I had captured your advice on that on the copper wiki at https://wiki.opnfv.org/display/copper/Joid 18:28:46 <bryan_att> https://www.irccloud.com/pastebin/qzoSyyVl/ 18:28:58 <bryan_att> is that incomplete? 18:29:15 <narindergupta> no its not an issue with reployment with maas 18:29:41 <narindergupta> this is issue where deployment.yaml missing from local directory which should get copied if does not exist 18:29:55 <bryan_att> ok 18:30:24 <narindergupta> it should be deployment.yaml file not envronments.yaml 18:30:38 <narindergupta> as environments.yaml file i am copying it anyway 18:30:58 <bryan_att> in my instructions on how to redeploy ? 18:32:34 <narindergupta> no in my script 18:32:57 <narindergupta> i usually copy everything now and your instructions are not needed anymore 18:33:03 <narindergupta> as i have patched it 18:33:09 <narindergupta> but missed deployment.yaml 18:33:51 <bryan_att> ok, so I can redeploy without having to copy anything... i.e. I can just run the deploy command on a clean clone of JOID and it will copy everything as needed. 18:34:32 <narindergupta> yes it should 18:34:55 <narindergupta> but due to this bug you need to check whether you have deployment.yaml in ~/.juju/ or not? 18:35:14 <narindergupta> if not then copy it for now meanwhile i am putting a patch in joid to fix it 18:37:14 <bryan_att> yes, it's there. I'll copy and redeploy 18:37:44 <narindergupta> yes you can do that 18:37:48 <narindergupta> that will fix it 18:37:56 <narindergupta> but i am putting a patch 18:38:01 <narindergupta> may be just wait 18:38:04 <narindergupta> for 5 minutes 18:38:09 <narindergupta> then you can reclone 18:40:46 <narindergupta> bryan_att, please review https://gerrit.opnfv.org/gerrit/20709 18:42:25 <narindergupta> bryan_att, please clone it now and retry 18:43:04 <bryan_att> will do 18:45:14 <narindergupta> thanks 19:13:54 <narindergupta> bryan_att, i think bundles.yaml should be created by now. Will you please check the latest one? 19:17:07 <bryan_att> ok, will check soon - at lunch 19:22:48 <narindergupta> ok no problem 19:35:55 <bryan_att> narindergupta: looks good 19:39:13 <narindergupta> cool 23:06:48 <bryan_att> narindergupta: it timed out with all the services in state "Waiting for agent initialization to finish" 23:11:37 <narindergupta> bryan_att, we need to see why there is no ip address defined. sometime in MAAS dhcp does not find free ip address 23:12:04 <narindergupta> please try to log into maas server and checks for /var/log/syslog 23:13:25 <bryan_att> what do I look for there? 23:13:37 <bryan_att> in maas.log 23:14:51 <bryan_att> I see in the MAAS UI that the nodes are in "failed deployment" 23:19:50 <bryan_att> The servers have IP addresses, I can ssh to them. 23:20:12 <bryan_att> I'll clean and try again. 23:21:12 <bryan_att> maybe I forgot to shut them off last time 23:24:05 <narindergupta> lets see look for no ip address 23:24:28 <narindergupta> i wish i get access to your system and see 23:24:42 <narindergupta> failed deployment may be 23:25:15 <narindergupta> so you need to release both the servers from maas and run ./cleanup.sh 23:30:44 <bryan_att> ok, did that, it's redeploying 23:31:07 <bryan_att> We can do a bluejeans session if you have an account there - I've used it with RedHat 13:31:21 <narindergupta> David_Orange, hi good morning. 13:32:44 <David_Orange> narindergupta: Hi 13:33:28 <narindergupta> David_Orange, yeah heat have issues with public apis. And we knows that it is not responding and working with those. 13:33:53 <narindergupta> so do we have the action plan or area we can concentrate to get it working? 13:33:58 <David_Orange> narindergupta: i will follow my test, i can not understand why it is failing 13:34:14 <David_Orange> i will try to remove the unit and reinstall it 13:34:30 <David_Orange> maybe the service installation order have an impact 13:35:46 <narindergupta> hun that wiered it should not be though 13:36:11 <David_Orange> it is only a guess, i will test it now 13:36:50 <David_Orange> but if the heat installation need to call some other apis, it may fails because it does not use internal urls 13:37:58 <narindergupta> for internal urls i could not options in the heat config. so heat has to modified for that 13:39:24 <David_Orange> as you have seen on the doc i sent, it is only a question of performance enhancement to use internal url, with the reverse proxy it will work on all configurations 13:40:00 <David_Orange> but we install it at the end, and mybe the heat post install procedure needs to access apis 13:40:22 <narindergupta> you mean creating domain setup? 13:40:27 <David_Orange> yes 13:40:38 <narindergupta> we can check that 13:40:46 <David_Orange> or some other scripts 13:40:54 <narindergupta> its part of action in charm 13:41:09 <David_Orange> it does 13:41:17 <narindergupta> oh ok 13:41:30 <David_Orange> from keystoneclient.v3 import client 13:41:37 <narindergupta> do you want me to avoid it at later stage 13:41:58 <David_Orange> avoid what ? 13:42:39 <narindergupta> create domain at the end 13:44:51 <David_Orange> at the end of the charm ? you can trigger that or not ? 13:45:18 <narindergupta> we can not trigger in charm installation since it is an action 13:45:25 <narindergupta> and we are using juju 13:45:36 <narindergupta> we are using juju-deployer 13:49:53 <narindergupta> David_Orange, yes domain-setup uses the api to create the role 13:50:40 <David_Orange> narindergupta: i removed heat unit and service, and tried to launch "juju-deployer -c bundles.yaml openstack-phase1" to reinstall it, but i got that: "cannot add unit 1/1 to service "heat": cannot add unit to service "heat": service is not alive'" 13:51:26 <narindergupta> can u run juju status heat 13:51:38 <narindergupta> it usually takes time to destroy the unit 13:51:59 <David_Orange> uju status heat 13:51:59 <David_Orange> environment: opnfv-orangepod1-maas 13:51:59 <David_Orange> machines: {} 13:51:59 <David_Orange> services: {} 13:51:59 <David_Orange> networks: 13:52:00 <David_Orange> subnet-1: 13:52:00 <David_Orange> provider-id: subnet-1 13:52:01 <David_Orange> cidr: 192.168.1.0/24 13:52:01 <David_Orange> subnet-2: 13:52:02 <David_Orange> provider-id: subnet-2 13:52:02 <David_Orange> cidr: 192.168.11.0/24 13:52:03 <David_Orange> subnet-3: 13:52:15 <narindergupta> ok can u run it now? 13:53:52 <narindergupta> and run this command juju-deployer -vW -d -t 7200 -r 5 -c bundles.yaml xenial-mitaka 13:54:14 <narindergupta> rather than openstack-phase1 as target 13:54:20 <David_Orange> ok 13:57:29 <narindergupta> this is the rc file heat uses 13:57:31 <narindergupta> cat /root/admin-openrc-v3 13:57:31 <narindergupta> # Service credentials for Keystone v3 13:57:31 <narindergupta> export OS_REGION_NAME=RegionOne 13:57:31 <narindergupta> export OS_PASSWORD=P5Sm66VmPNK2WtBmsk6drzZKZFrPwgYjc5ZfcVBf33KrYZYpsPgd7Hc6xnLVCYFq 13:57:31 <narindergupta> export OS_AUTH_URL=http://10.6.1.199:35357/v3 13:57:32 <narindergupta> export OS_USERNAME=heat-cfn_heat 13:57:34 <narindergupta> export OS_TENANT_NAME=services 13:57:36 <narindergupta> export OS_PROJECT_DOMAIN_ID=default 13:57:38 <narindergupta> export OS_USER_DOMAIN_ID=default 13:57:42 <narindergupta> export OS_PROJECT_NAME=services 13:57:44 <narindergupta> export OS_IDENTITY_API_VERSION=3 13:58:06 <David_Orange> same answer 13:58:07 <David_Orange> cannot add unit 1/1 to service "heat": cannot add unit to service "heat": service is not alive', 13:58:20 <narindergupta> it should not add unit 13:58:43 <narindergupta> it should create new unit of heat 13:59:23 <narindergupta> juju deploy heat local:xenial/heat 13:59:52 <narindergupta> then we can try the juju-deployer 13:59:55 <David_Orange> error: invalid service name "local:xenial/heat" 14:00:43 <narindergupta> hun i need to think about it what i tried last time 14:01:23 <David_Orange> ok 14:01:38 <narindergupta> juju deploy local:xenial/heat 14:02:29 <David_Orange> ERROR cannot add service "heat": service already exists 14:02:37 <David_Orange> but still got: 14:02:37 <David_Orange> juju status heat 14:02:37 <David_Orange> environment: opnfv-orangepod1-maas 14:02:37 <David_Orange> machines: {} 14:02:37 <David_Orange> services: {} 14:02:53 <narindergupta> hun looks like clean up few issues then 14:03:33 <narindergupta> you are on pod1 right? 14:03:45 <David_Orange> yes 14:04:26 <narindergupta> ok there are few error in rabbitmq 14:04:43 <narindergupta> gateway service is not running says blocked 14:04:53 <David_Orange> can you go to byobu 14:05:18 <David_Orange> it got an erro on gateway, the public ip was not set 14:05:32 <narindergupta> sure 14:05:51 <narindergupta> ok i am at byobu 14:07:18 <David_Orange> the ip was not set on gateway but did not get an error on jenkins 14:07:20 <David_Orange> + juju run --unit nodes/0 'sudo ip a a 10.0.1.4/24 dev br-ex' + juju run --unit nodes/0 'sudo ip l set dev br-ex up' 14:09:19 <narindergupta> i think in your lab we do not set ip on external network through MAAS 14:09:25 <narindergupta> cani have access to MAAS 14:09:53 <narindergupta> currently username password to access site is not working 14:09:57 <David_Orange> it set it through openstack.sh 14:10:20 <David_Orange> ok i check that 14:10:53 <David_Orange> the public ip is set on line 138 of openstack.sh 14:12:21 <narindergupta> David_Orange, if you need ip then we can do it through maas itself like we do it for other labs 14:12:44 <narindergupta> there is no harm in defining the dhcp on this interface through maas 14:13:10 <narindergupta> oh but we do not know the what ip to set for proxy 14:13:20 <narindergupta> never mind 14:14:16 <David_Orange> yes :) 14:14:23 <David_Orange> it needs to be fixed 14:17:51 <narindergupta> rabbitmq throwing error on leader status change 14:17:55 <narindergupta> can i deploy? 14:18:15 <David_Orange> yes 14:18:34 <David_Orange> with jenkins ? 14:18:47 <narindergupta> yes 14:18:53 <David_Orange> good for me 14:20:33 <narindergupta> ok redeploying it now 14:21:04 <narindergupta> did you shutdown bootstrap node? 14:24:43 <narindergupta> was there any change in ssh config 14:24:59 <narindergupta> i can connct to qemu through sh keys] 14:28:09 <narindergupta> looks like it redeployed yesterday as well 15:01:38 <David_Orange> sorry, i was in a meeting. i redeployed it yesterday 15:02:30 <narindergupta> David_Orange, ok i have redploy maas as bootstrap had some issue 15:02:42 <David_Orange> ok 18:43:11 <narindergupta> bryan_att, any update on deployment? 13:15:15 <bryan_att> narindergupta: the deploy worked although I had to run the the command manually at the end " ./openstack.sh nosdn attvirpod1 xenial mitaka". I can send you the deploy log if needed. 13:15:54 <narindergupta> bryan_att, please send me as deployment should run the openstack.sh 13:16:30 <narindergupta> bryan_att, but heat domain set up ran with juju? 13:18:02 <bryan_att> https://usercontent.irccloud-cdn.com/file/73LpBM1k/deploy.log 13:18:23 <bryan_att> Not sure about the heat domain yet. I haven't run my Tacker test yet. 13:33:20 <narindergupta> bryan_att, looks like external network was not created 13:33:22 <narindergupta> public endpoint for network service in RegionOne region not found 13:34:20 <narindergupta> does juju status shows any error? if not then please check bundles.yaml also. whats the region name? 13:34:24 <bryan_att> the net was created when I re-ran the openstack script 13:34:43 <bryan_att> RegionOne 13:35:06 <bryan_att> No errors in JuJu 13:35:50 <narindergupta> ok looks like timing issue. 13:36:24 <narindergupta> i can see after juju-dpeloyer completes your timeout of 30 seconds expires 13:36:43 <narindergupta> so either i have increase the time out for you here 13:36:51 <narindergupta> or find some alternative. 13:37:00 <narindergupta> before i runs openstack.sh 13:37:47 <narindergupta> will you please run juju action do heat/0 domain-setup 13:37:54 <narindergupta> again before you can start tacker 03:02:06 <Jing> narinder`: ping 03:16:58 <Jing> Hi! joid guys. I am a yardsticker and I am working on fixing the yardstick CI job failure in joid-os-nosdn-lxd-ha-baremetal-daily-colorado. I have a few questions. I have figured out a raw format image is needed for nova-lxd. What other requirements does lxd scenario needs? Can anyone help? 12:32:42 <OPNFV-Gerrit-Bot> Narinder Gupta proposed joid: modified the default URL to Boron RC3.5 https://gerrit.opnfv.org/gerrit/21015 12:32:47 <OPNFV-Gerrit-Bot> A comment has been added to a proposed change to joid: modified the default URL to Boron RC3.5 https://gerrit.opnfv.org/gerrit/21015 12:33:05 <OPNFV-Gerrit-Bot> A comment has been added to a proposed change to joid: modified the default URL to Boron RC3.5 https://gerrit.opnfv.org/gerrit/21015 12:33:18 <OPNFV-Gerrit-Bot> A comment has been added to a proposed change to joid: modified the default URL to Boron RC3.5 https://gerrit.opnfv.org/gerrit/21015 12:33:21 <OPNFV-Gerrit-Bot> Merged joid: modified the default URL to Boron RC3.5 https://gerrit.opnfv.org/gerrit/21015 14:19:33 <narindergupta> David_Orange, hi how r u? 14:19:59 <David_Orange> Hi narinder, fine and you ? 15:18:55 <bryan_att> narindergupta: ping 15:21:05 <bryan_att> I have an issue with the JOID deployment of Heat, that is not occuring with Apex. What version of Heat are you deploying? 15:21:08 <narindergupta> bryan_att, how are you 15:21:21 <bryan_att> fine, just have an issue with Heat. 15:21:27 <narindergupta> bryan_att, let me check with charm 15:21:46 <narindergupta> bryan_att, let me check 15:22:52 <narindergupta> it is 1.6.0 15:23:01 <narindergupta> 1:6.0.0-0ubuntu1.1 15:23:02 <bryan_att> Heat on JOID rejects the stack CP1.properties 'port_security_enabled' flag (Apex Heat accepts this) , resulting in "Resource CREATE failed: BadRequest: resources.CP1: Unrecognized attribute(s) 'port_security_enabled' Neutron server " 15:23:25 <bryan_att> OK, let me check with Apex on the version they are deploying. 15:25:27 <narindergupta> bryan_att, ok 15:32:12 <David_Orange> narindergupta: just a little status on heat, i have to go: i tried your patch https://www.irccloud.com/pastebin/y4ub3ohr/ but i got "message": "You are not authorized to use create." error message, so i will follow my tests tomorrow. FYI i found a bug for heat+publicapi, i will push the patch tomorrow 15:32:45 <narindergupta> ok sounds good to me 18:10:46 <bryan_att> narindergupta: Apex is deploying Heat version 6.0.1-0.20160820004902.13aaa7e.el7.centos. How does that compare to 1:6.0.0-0ubuntu1.1 ? Both are used in OPNFV Colorado, but I'm seeing differences in behavior. How would I find out what version is in these packages? 18:14:08 <narindergupta> Ubuntu heat packages are at lacunhpad https://launchpad.net/ubuntu/+source/heat 18:14:17 <bryan_att> narindergupta: Does JOID enable the port_security extension as mentioned in prerequisites section of the Tacker manual installation guide? Step 4: http://docs.openstack.org/developer/tacker/install/manual_installation.html#pre-requisites 18:19:29 <narindergupta> bryan_att, by default it is disabled in neutron-api charm 18:20:17 <bryan_att> narindergupta: this needs to be the setting - this is what Apex deploys and works. https://www.irccloud.com/pastebin/xOnqwuEI/ 18:20:37 <narindergupta> but it is needed you can enable it using juju set neutron-api enable-ml2-port-security=true 18:20:38 <bryan_att> "extension_drivers =port_security" at least 18:20:59 <narindergupta> by default this feature is false 18:21:29 <bryan_att> OK, for Colorado I can do that as part of my tests setup I guess - but for Danube can we enable it by default? It's required for Tacker to work. 18:21:55 <bryan_att> And I'd like it to be enabled by default in Colorado if that doesn't break something else. 18:22:16 <narindergupta> bryan_att, we need to see the implications on other tests if we enable it. So just made a request in joid and we will enable it in D release 18:22:56 <narindergupta> bryan_att, currently C release it would be difficult as we are releasing it soon. 18:23:24 <narindergupta> and everything is stable. I can try for experimental basis in master branch though 03:15:07 <Jing> Hi! joid guys. I am a yardsticker and I am working on fixing the yardstick CI job failure in joid-os-nosdn-lxd-ha-baremetal-daily-colorado. I have a few questions. I have figured out a raw format image is needed for nova-lxd. What other requirements does lxd scenario needs? Can anyone help? 03:47:13 <narindergupta> Jing, hi jing as you know LXD is contianer based hypervisor. So all images on the nova should be lxd or lxc based. 03:48:09 <narindergupta> and any application or test you run on the container should be same. As after initiating image on nova you should able to ssh into it and run any commands like apt-get install etc etc... 03:49:08 <narindergupta> Jing, what else information do you need? As part of LXd deployment installer upload the lxc images for cirros and ubuntu lxd images as well 03:49:18 <narindergupta> Jing, what else information do you need? As part of LXd deployment installer upload the lxc images for cirros and ubuntu lxd images as well 03:49:52 <narindergupta> please let me know in case you need any specific information on nova-lxd driver and images apart from what i provided 04:12:01 <narindergupta> #opnfv-onos 04:33:47 <narindergupta> #endmeting 04:34:22 <narindergupta> Jing, do you have any other query on joid? 07:19:25 <Jing> narindergupta: hi, Just for clarity, in lxd we need to used lxc images instead of normal images? What is the difference between them? 16:02:56 <collabot> arturt: Error: Can't start another meeting, one is in progress. Use #endmeeting first. 16:03:02 <arturt> #stopmeeting 16:03:15 <arturt> #endmeeting