16:05:03 <narindergupta> #startmeeting JOID
16:05:03 <collabot> Meeting started Wed Sep  7 16:05:03 2016 UTC.  The chair is narindergupta. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:05:03 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:05:03 <collabot> The meeting name has been set to 'joid'
16:15:19 <narindergupta> #info https://www.irccloud.com/pastebin/y4ub3ohr/
16:19:03 <narindergupta> #info include the directories and files will be removed in D release which we kept it for legacy purpose
17:53:33 <bryan_att> narindergupta: ping
17:53:50 <narindergupta> bryan_att, hi how are you?
17:54:31 <bryan_att> fine, just an issue with JOID deploy, cloned yesterday; tried twice and it ends with a keystone error
17:54:49 <bryan_att> https://www.irccloud.com/pastebin/dKStScXQ/
17:55:01 <narindergupta> is it master build or stable ?
17:55:06 <bryan_att> Master
17:55:17 <bryan_att> Should I be using stable?
17:55:44 <narindergupta> it should not matter as both are same
17:55:56 <narindergupta> and i am hopinh juju status shows no error?
17:56:03 <bryan_att> JuJu status looks fine
17:56:15 <narindergupta> this is problem Unable to establish connection to http://api.None:5000/v2.0/tokens
17:56:22 <bryan_att> yes
17:56:50 <narindergupta> please send meyasml files
17:56:56 <narindergupta> bundles.yaml first?
17:57:00 <bryan_att> ok
17:57:08 <narindergupta> also your labconfig.yaml
17:57:19 <bryan_att> https://usercontent.irccloud-cdn.com/file/uCkPZydN/bundles.yaml
17:57:42 <bryan_att> https://usercontent.irccloud-cdn.com/file/LuHLdupX/labconfig.yaml
17:59:29 <narindergupta> ok in your bundles.yaml we have os-public-hostname: api.None
18:00:17 <narindergupta> bryan_att, please send me deployconfig.yaml as well
18:00:33 <bryan_att> https://usercontent.irccloud-cdn.com/file/SayJNE6E/deployconfig.yaml
18:09:54 <narindergupta> will you please run this command python genBundle.py  -l deployconfig.yaml  -s os-nosdn-nofeature-nonha > bundles.yaml
18:10:01 <narindergupta> and send me bundles.yaml
18:10:26 <narindergupta> it looks like some issue in your case while in my case almost similar value no issues
18:10:41 <bryan_att> https://usercontent.irccloud-cdn.com/file/73EATKP4/bundles.yaml
18:12:28 <narindergupta> bryan_att, in this file looks like everything is ok
18:12:39 <narindergupta> somehow your bundles.yaml is wrong
18:12:58 <bryan_att> is it generated upon the deploy?
18:13:02 <narindergupta> yes
18:13:13 <narindergupta> and placed under joid/ci
18:13:22 <bryan_att> what generates it?
18:13:45 <narindergupta> which option r u using in -f here?
18:14:04 <bryan_att> you mean in the deploy command?
18:14:10 <narindergupta> yes
18:14:21 <bryan_att> ./deploy.sh -o mitaka -s nosdn -t nonha -l attvirpod1
18:14:34 <bryan_att> not using -f
18:14:46 <narindergupta> that should not add api then
18:15:14 <narindergupta> will you please send me console log for deployment?
18:15:18 <bryan_att> sure
18:15:37 <narindergupta> it seems code is working and manually i verified it
18:15:51 <narindergupta> so we need to find out the error which is causing it
18:19:15 <bryan_att> https://usercontent.irccloud-cdn.com/file/F1ImkPIr/deploy.log
18:19:49 <bryan_att> I see "+ keystone_PUBENDPOINT=api.None"
18:20:26 <bryan_att> right after https://www.irccloud.com/pastebin/uCP3LUwt/
18:21:26 <narindergupta> yeah here is the problem python genBundle.py -l deployconfig.yaml -s os-nosdn-none-nonha-xenial_mitaka_publicapi
18:21:33 <narindergupta> publicapi should not be added
18:23:19 <bryan_att> so I can patch it locally and retry? what do I change? or should I wait for a patch?
18:26:30 <narindergupta> bryan_att, do you have deployment.yaml in your local directory?
18:27:12 <bryan_att> in joid/ci ? no
18:27:42 <narindergupta> bryan_att, ok that was root cause of issue. I will work on it meanwhile please copy that from ~/.juju/
18:27:59 <narindergupta> and cleanup and restart the deployment
18:28:36 <bryan_att> ok, maybe this was an issue with how I redployed without redeploying maas  I had captured your advice on that on the copper wiki at https://wiki.opnfv.org/display/copper/Joid
18:28:46 <bryan_att> https://www.irccloud.com/pastebin/qzoSyyVl/
18:28:58 <bryan_att> is that incomplete?
18:29:15 <narindergupta> no its not an issue with reployment with maas
18:29:41 <narindergupta> this is issue where deployment.yaml  missing from local directory which should get copied if does not exist
18:29:55 <bryan_att> ok
18:30:24 <narindergupta> it should be deployment.yaml file not envronments.yaml
18:30:38 <narindergupta> as environments.yaml file i am copying it anyway
18:30:58 <bryan_att> in my instructions on how to redeploy ?
18:32:34 <narindergupta> no in my script
18:32:57 <narindergupta> i usually copy everything now and your instructions are not needed anymore
18:33:03 <narindergupta> as i have patched it
18:33:09 <narindergupta> but missed deployment.yaml
18:33:51 <bryan_att> ok, so I can redeploy without having to copy anything... i.e. I can just run the deploy command on a clean clone of JOID and it will copy everything as needed.
18:34:32 <narindergupta> yes it should
18:34:55 <narindergupta> but due to this bug you need to check whether you have deployment.yaml in ~/.juju/ or not?
18:35:14 <narindergupta> if not then copy it for now meanwhile i am putting a patch in joid to fix it
18:37:14 <bryan_att> yes, it's there. I'll copy and redeploy
18:37:44 <narindergupta> yes you can do that
18:37:48 <narindergupta> that will fix it
18:37:56 <narindergupta> but i am putting a patch
18:38:01 <narindergupta> may be just wait
18:38:04 <narindergupta> for 5 minutes
18:38:09 <narindergupta> then you can reclone
18:40:46 <narindergupta> bryan_att, please review https://gerrit.opnfv.org/gerrit/20709
18:42:25 <narindergupta> bryan_att, please clone it now and retry
18:43:04 <bryan_att> will do
18:45:14 <narindergupta> thanks
19:13:54 <narindergupta> bryan_att, i think bundles.yaml should be created by now. Will you please check the latest one?
19:17:07 <bryan_att> ok, will check soon - at lunch
19:22:48 <narindergupta> ok no problem
19:35:55 <bryan_att> narindergupta: looks good
19:39:13 <narindergupta> cool
23:06:48 <bryan_att> narindergupta: it timed out with all the services in state "Waiting for agent initialization to finish"
23:11:37 <narindergupta> bryan_att, we need to see why there is no ip address defined. sometime in MAAS dhcp does not find free ip address
23:12:04 <narindergupta> please try to log into maas server and checks for /var/log/syslog
23:13:25 <bryan_att> what do I look for there?
23:13:37 <bryan_att> in maas.log
23:14:51 <bryan_att> I see in the MAAS UI that the nodes are in "failed deployment"
23:19:50 <bryan_att> The servers have IP addresses, I can ssh to them.
23:20:12 <bryan_att> I'll clean and try again.
23:21:12 <bryan_att> maybe I forgot to shut them off last time
23:24:05 <narindergupta> lets see look for no ip address
23:24:28 <narindergupta> i wish i get access to your system and see
23:24:42 <narindergupta> failed deployment may be
23:25:15 <narindergupta> so you need to release both the servers from maas and run ./cleanup.sh
23:30:44 <bryan_att> ok, did that, it's redeploying
23:31:07 <bryan_att> We can do a bluejeans session if you have an account there - I've used it with RedHat
13:31:21 <narindergupta> David_Orange, hi good morning.
13:32:44 <David_Orange> narindergupta: Hi
13:33:28 <narindergupta> David_Orange, yeah heat have issues with public apis. And we knows that it is not responding and working with those.
13:33:53 <narindergupta> so do we have the action plan or area we can concentrate to get it working?
13:33:58 <David_Orange> narindergupta: i will follow my test, i can not understand why it is failing
13:34:14 <David_Orange> i will try to remove the unit and reinstall it
13:34:30 <David_Orange> maybe the service installation order have an impact
13:35:46 <narindergupta> hun that wiered it should not be though
13:36:11 <David_Orange> it is only a guess, i will test it now
13:36:50 <David_Orange> but if the heat installation need to call some other apis, it may fails because it does not use internal urls
13:37:58 <narindergupta> for internal urls i could not options in the heat config. so heat has to modified for that
13:39:24 <David_Orange> as you have seen on the doc i sent, it is only a question of performance enhancement to use internal url, with the reverse proxy it will work on all configurations
13:40:00 <David_Orange> but we install it at the end, and mybe the heat post install procedure needs to access apis
13:40:22 <narindergupta> you mean creating domain setup?
13:40:27 <David_Orange> yes
13:40:38 <narindergupta> we can check that
13:40:46 <David_Orange> or some other scripts
13:40:54 <narindergupta> its part of action in charm
13:41:09 <David_Orange> it does
13:41:17 <narindergupta> oh ok
13:41:30 <David_Orange> from keystoneclient.v3 import client
13:41:37 <narindergupta> do you want me to avoid it at later stage
13:41:58 <David_Orange> avoid what ?
13:42:39 <narindergupta> create domain at the end
13:44:51 <David_Orange> at the end of the charm ? you can trigger that or not ?
13:45:18 <narindergupta> we can not trigger in charm installation since it is an action
13:45:25 <narindergupta> and we are using juju
13:45:36 <narindergupta> we are using juju-deployer
13:49:53 <narindergupta> David_Orange, yes domain-setup uses the api to create the role
13:50:40 <David_Orange> narindergupta: i removed heat unit and service, and tried to launch "juju-deployer -c bundles.yaml openstack-phase1" to reinstall it, but i got that: "cannot add unit 1/1 to service "heat": cannot add unit to service "heat": service is not alive'"
13:51:26 <narindergupta> can u run juju status heat
13:51:38 <narindergupta> it usually takes time to destroy the unit
13:51:59 <David_Orange> uju status heat
13:51:59 <David_Orange> environment: opnfv-orangepod1-maas
13:51:59 <David_Orange> machines: {}
13:51:59 <David_Orange> services: {}
13:51:59 <David_Orange> networks:
13:52:00 <David_Orange> subnet-1:
13:52:00 <David_Orange> provider-id: subnet-1
13:52:01 <David_Orange> cidr: 192.168.1.0/24
13:52:01 <David_Orange> subnet-2:
13:52:02 <David_Orange> provider-id: subnet-2
13:52:02 <David_Orange> cidr: 192.168.11.0/24
13:52:03 <David_Orange> subnet-3:
13:52:15 <narindergupta> ok can u run it now?
13:53:52 <narindergupta> and run this command juju-deployer -vW -d -t 7200 -r 5 -c bundles.yaml xenial-mitaka
13:54:14 <narindergupta> rather than openstack-phase1 as target
13:54:20 <David_Orange> ok
13:57:29 <narindergupta> this is the rc file heat uses
13:57:31 <narindergupta> cat /root/admin-openrc-v3
13:57:31 <narindergupta> # Service credentials for Keystone v3
13:57:31 <narindergupta> export OS_REGION_NAME=RegionOne
13:57:31 <narindergupta> export OS_PASSWORD=P5Sm66VmPNK2WtBmsk6drzZKZFrPwgYjc5ZfcVBf33KrYZYpsPgd7Hc6xnLVCYFq
13:57:31 <narindergupta> export OS_AUTH_URL=http://10.6.1.199:35357/v3
13:57:32 <narindergupta> export OS_USERNAME=heat-cfn_heat
13:57:34 <narindergupta> export OS_TENANT_NAME=services
13:57:36 <narindergupta> export OS_PROJECT_DOMAIN_ID=default
13:57:38 <narindergupta> export OS_USER_DOMAIN_ID=default
13:57:42 <narindergupta> export OS_PROJECT_NAME=services
13:57:44 <narindergupta> export OS_IDENTITY_API_VERSION=3
13:58:06 <David_Orange> same answer
13:58:07 <David_Orange> cannot add unit 1/1 to service "heat": cannot add unit to service "heat": service is not alive',
13:58:20 <narindergupta> it should not add unit
13:58:43 <narindergupta> it should create new unit of heat
13:59:23 <narindergupta> juju deploy heat local:xenial/heat
13:59:52 <narindergupta> then we can try the juju-deployer
13:59:55 <David_Orange> error: invalid service name "local:xenial/heat"
14:00:43 <narindergupta> hun i need to think about it what i tried last time
14:01:23 <David_Orange> ok
14:01:38 <narindergupta> juju deploy local:xenial/heat
14:02:29 <David_Orange> ERROR cannot add service "heat": service already exists
14:02:37 <David_Orange> but still got:
14:02:37 <David_Orange> juju status heat
14:02:37 <David_Orange> environment: opnfv-orangepod1-maas
14:02:37 <David_Orange> machines: {}
14:02:37 <David_Orange> services: {}
14:02:53 <narindergupta> hun looks like clean up few issues then
14:03:33 <narindergupta> you are on pod1 right?
14:03:45 <David_Orange> yes
14:04:26 <narindergupta> ok there are few error in rabbitmq
14:04:43 <narindergupta> gateway service is not running says blocked
14:04:53 <David_Orange> can you go to byobu
14:05:18 <David_Orange> it got an erro on gateway, the public ip was not set
14:05:32 <narindergupta> sure
14:05:51 <narindergupta> ok i am at byobu
14:07:18 <David_Orange> the ip was not set on gateway but did not get an error on jenkins
14:07:20 <David_Orange> + juju run --unit nodes/0 'sudo ip a a  10.0.1.4/24 dev br-ex' + juju run --unit nodes/0 'sudo ip l set dev br-ex up'
14:09:19 <narindergupta> i think in your lab we do not set ip on external network through MAAS
14:09:25 <narindergupta> cani have access to MAAS
14:09:53 <narindergupta> currently username password to access site is not working
14:09:57 <David_Orange> it set it through openstack.sh
14:10:20 <David_Orange> ok i check that
14:10:53 <David_Orange> the public ip is set on line 138 of openstack.sh
14:12:21 <narindergupta> David_Orange, if you need ip then we can do it through maas itself like we do it for other labs
14:12:44 <narindergupta> there is no harm in defining the dhcp on this interface through maas
14:13:10 <narindergupta> oh but we do not know the what ip to set for proxy
14:13:20 <narindergupta> never mind
14:14:16 <David_Orange> yes :)
14:14:23 <David_Orange> it needs to be fixed
14:17:51 <narindergupta> rabbitmq throwing error on leader status change
14:17:55 <narindergupta> can i deploy?
14:18:15 <David_Orange> yes
14:18:34 <David_Orange> with jenkins ?
14:18:47 <narindergupta> yes
14:18:53 <David_Orange> good for me
14:20:33 <narindergupta> ok redeploying it now
14:21:04 <narindergupta> did you shutdown bootstrap node?
14:24:43 <narindergupta> was there any change in ssh config
14:24:59 <narindergupta> i can connct to qemu through sh keys]
14:28:09 <narindergupta> looks like it redeployed yesterday as well
15:01:38 <David_Orange> sorry, i was in a meeting. i redeployed it yesterday
15:02:30 <narindergupta> David_Orange, ok i have redploy maas as bootstrap had some issue
15:02:42 <David_Orange> ok
18:43:11 <narindergupta> bryan_att, any update on deployment?
13:15:15 <bryan_att> narindergupta: the deploy worked although I had to run the the command manually at the end " ./openstack.sh nosdn attvirpod1 xenial mitaka". I can send you the deploy log if needed.
13:15:54 <narindergupta> bryan_att, please send me as deployment should run the openstack.sh
13:16:30 <narindergupta> bryan_att, but heat domain set up ran with juju?
13:18:02 <bryan_att> https://usercontent.irccloud-cdn.com/file/73LpBM1k/deploy.log
13:18:23 <bryan_att> Not sure about the heat domain yet. I haven't run my Tacker test yet.
13:33:20 <narindergupta> bryan_att, looks like external network was not created
13:33:22 <narindergupta> public endpoint for network service in RegionOne region not found
13:34:20 <narindergupta> does juju status shows any error? if not then please check bundles.yaml also. whats the region name?
13:34:24 <bryan_att> the net was created when I re-ran the openstack script
13:34:43 <bryan_att> RegionOne
13:35:06 <bryan_att> No errors in JuJu
13:35:50 <narindergupta> ok looks like timing issue.
13:36:24 <narindergupta> i can see after juju-dpeloyer completes your timeout of 30 seconds expires
13:36:43 <narindergupta> so either i have increase the time out for you here
13:36:51 <narindergupta> or find some alternative.
13:37:00 <narindergupta> before i runs openstack.sh
13:37:47 <narindergupta> will you please run juju action do heat/0 domain-setup
13:37:54 <narindergupta> again before you can start tacker
03:02:06 <Jing> narinder`: ping
03:16:58 <Jing> Hi! joid guys. I am a yardsticker and I am working on fixing the yardstick CI job failure in joid-os-nosdn-lxd-ha-baremetal-daily-colorado. I have a few questions. I have figured out a raw format image is needed for nova-lxd. What other requirements does lxd scenario needs? Can anyone help?
12:32:42 <OPNFV-Gerrit-Bot> Narinder Gupta proposed joid: modified the default URL to Boron RC3.5  https://gerrit.opnfv.org/gerrit/21015
12:32:47 <OPNFV-Gerrit-Bot> A comment has been added to a proposed change to joid: modified the default URL to Boron RC3.5  https://gerrit.opnfv.org/gerrit/21015
12:33:05 <OPNFV-Gerrit-Bot> A comment has been added to a proposed change to joid: modified the default URL to Boron RC3.5  https://gerrit.opnfv.org/gerrit/21015
12:33:18 <OPNFV-Gerrit-Bot> A comment has been added to a proposed change to joid: modified the default URL to Boron RC3.5  https://gerrit.opnfv.org/gerrit/21015
12:33:21 <OPNFV-Gerrit-Bot> Merged joid: modified the default URL to Boron RC3.5  https://gerrit.opnfv.org/gerrit/21015
14:19:33 <narindergupta> David_Orange, hi how r u?
14:19:59 <David_Orange> Hi narinder, fine and you ?
15:18:55 <bryan_att> narindergupta: ping
15:21:05 <bryan_att> I have an issue with the JOID deployment of Heat, that is not occuring with Apex. What version of Heat are you deploying?
15:21:08 <narindergupta> bryan_att, how are you
15:21:21 <bryan_att> fine, just have an issue with Heat.
15:21:27 <narindergupta> bryan_att, let me check with charm
15:21:46 <narindergupta> bryan_att, let me check
15:22:52 <narindergupta> it is 1.6.0
15:23:01 <narindergupta> 1:6.0.0-0ubuntu1.1
15:23:02 <bryan_att> Heat on JOID rejects the stack CP1.properties 'port_security_enabled' flag (Apex Heat accepts this) , resulting in "Resource CREATE failed: BadRequest: resources.CP1: Unrecognized attribute(s) 'port_security_enabled' Neutron server "
15:23:25 <bryan_att> OK, let me check with Apex on the version they are deploying.
15:25:27 <narindergupta> bryan_att, ok
15:32:12 <David_Orange> narindergupta: just a little status on heat, i have to go: i tried your patch  https://www.irccloud.com/pastebin/y4ub3ohr/ but i got "message": "You are not authorized to use create." error message, so i will follow my tests tomorrow. FYI i found a bug for heat+publicapi, i will push the patch tomorrow
15:32:45 <narindergupta> ok sounds good to me
18:10:46 <bryan_att> narindergupta: Apex is deploying Heat version 6.0.1-0.20160820004902.13aaa7e.el7.centos. How does that compare to 1:6.0.0-0ubuntu1.1 ? Both are used in OPNFV Colorado, but I'm seeing differences in behavior. How would I find out what version is in these packages?
18:14:08 <narindergupta> Ubuntu heat packages are at lacunhpad https://launchpad.net/ubuntu/+source/heat
18:14:17 <bryan_att> narindergupta: Does JOID enable the port_security extension as mentioned in prerequisites section of the Tacker manual installation guide? Step 4: http://docs.openstack.org/developer/tacker/install/manual_installation.html#pre-requisites
18:19:29 <narindergupta> bryan_att, by default it is disabled in neutron-api charm
18:20:17 <bryan_att> narindergupta: this needs to be the setting - this is what Apex deploys and works. https://www.irccloud.com/pastebin/xOnqwuEI/
18:20:37 <narindergupta> but it is needed you can enable it using juju set neutron-api enable-ml2-port-security=true
18:20:38 <bryan_att> "extension_drivers =port_security" at least
18:20:59 <narindergupta> by default this feature is false
18:21:29 <bryan_att> OK, for Colorado I can do that as part of my tests setup I guess - but for Danube can we enable it by default? It's required for Tacker to work.
18:21:55 <bryan_att> And I'd like it to be enabled by default in Colorado if that doesn't break something else.
18:22:16 <narindergupta> bryan_att, we need to see the implications on other tests if we enable it. So just made a request in joid and we will enable it in D release
18:22:56 <narindergupta> bryan_att, currently C release it would be difficult as we are releasing it soon.
18:23:24 <narindergupta> and everything is stable. I can try for experimental basis in master branch though
03:15:07 <Jing> Hi! joid guys. I am a yardsticker and I am working on fixing the yardstick CI job failure in joid-os-nosdn-lxd-ha-baremetal-daily-colorado. I have a few questions. I have figured out a raw format image is needed for nova-lxd. What other requirements does lxd scenario needs? Can anyone help?
03:47:13 <narindergupta> Jing, hi jing as you know LXD is contianer based hypervisor. So all images on the nova should be lxd or lxc based.
03:48:09 <narindergupta> and any application or test you run on the container should be same. As after initiating image on nova you should able to ssh into it and run any commands like apt-get install etc etc...
03:49:08 <narindergupta> Jing, what else information do you need? As part of LXd deployment installer upload the lxc images for cirros and ubuntu lxd images as well
03:49:18 <narindergupta> Jing, what else information do you need? As part of LXd deployment installer upload the lxc images for cirros and ubuntu lxd images as well
03:49:52 <narindergupta> please let me know in case you need any specific information on nova-lxd driver and images apart from what i provided
04:12:01 <narindergupta> #opnfv-onos
04:33:47 <narindergupta> #endmeting
04:34:22 <narindergupta> Jing, do you have any other query on joid?
07:19:25 <Jing> narindergupta: hi, Just for clarity, in lxd we need to used lxc images instead of normal images? What is the difference between them?
16:02:56 <collabot> arturt: Error: Can't start another meeting, one is in progress.  Use #endmeeting first.
16:03:02 <arturt> #stopmeeting
16:03:15 <arturt> #endmeeting