07:10:33 <joehuang> #startmeeting multisite
07:10:33 <collabot> Meeting started Thu Aug  4 07:10:33 2016 UTC.  The chair is joehuang. Information about MeetBot at http://wiki.debian.org/MeetBot.
07:10:33 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
07:10:33 <collabot> The meeting name has been set to 'multisite'
07:10:44 <sorantis> #info dimitri
07:10:51 <joehuang> #info joehuang
07:10:56 <May-meimei> #info meimei
07:11:11 <May-meimei> I talked to Ashish while ago
07:11:44 <joehuang> how about the functest in CI
07:11:57 <joehuang> Hello, Ashish
07:12:03 <May-meimei> he is trouble-shooting the access problem
07:12:14 <SAshish> Hi Joe, By mistake I connected to some other group
07:12:23 <SAshish> yeah, Hi All
07:12:38 <joehuang> no problem, welcome back :)
07:12:51 <SAshish> we need to resolve one the access problem, then tempest can work
07:13:08 <SAshish> other things are happening with tempest, where we create user/project for kb
07:13:13 <joehuang> do you mean the 8118 port can't be accessed in the container
07:13:35 <SAshish> I think, the mapping from public ip to internal ip is not proper
07:13:46 <SAshish> 192. internal Ip
07:13:52 <SAshish> 172. is public ip
07:14:00 <joehuang> is this the mapping by Fuel?
07:14:09 <SAshish> HA in fuel
07:14:35 <SAshish> I have seen configuration for the mapping only in HA configuration files
07:14:39 <sorantis> try grepping on the ip and port number of nova
07:14:39 <joehuang> Is kingbird running with Nova in same node?
07:14:43 <SAshish> yes
07:14:51 <sorantis> see where it’s set and make sure that kb follows the same
07:15:03 <SAshish> it is same
07:15:11 <sorantis> start with the /etc
07:15:16 <SAshish> iptables/haconfigurations/kb.conf
07:15:23 <SAshish> all are same
07:15:26 <sorantis> is it only this?
07:15:38 <SAshish> I think so
07:15:50 <joehuang> I have an idea
07:15:55 <SAshish> okay
07:16:18 <joehuang> change the port of kingbird to one of other service port which works in functest
07:16:55 <SAshish> 1 sec
07:16:56 <SAshish> okay
07:17:21 <SAshish> http://192.168.0.2:9292
07:17:25 <SAshish> this url is registered in keystone for glance
07:18:09 <SAshish> but the bind host is 192.168.0.5
07:18:15 <SAshish> so ours is also same
07:18:26 <SAshish> bind host in kb.conf is 192.168.0.5
07:18:46 <SAshish> url in keystone is http://192.168.0.2:8118/v1.0
07:18:49 <SAshish> now
07:19:21 <joehuang> really strange, I have no idea about Fuel's setting why.
07:19:22 <SAshish> from the controller glance is accessible with both the addresses
07:19:35 <SAshish> i mean, telnet 192.168.0.5 8774
07:19:36 <SAshish> works
07:19:41 <SAshish> telnet 192.168.0.2 8774
07:19:42 <SAshish> works
07:19:57 <SAshish> but for kingbird it doesn not, it works only with the bind host registered in kb.conf
07:20:12 <SAshish> not the url mentioned in keystone
07:20:20 <joehuang> you can use 0.0.0.0 for bind host
07:20:30 <joehuang> other wise I guess you can't access it
07:20:49 <joehuang> for 0.0.0.0 means all IP address for this host works for the service
07:21:13 <SAshish> in all other configurations it is the ip
07:21:36 <joehuang> how many ip address the host have?
07:21:40 <SAshish> so kept the same for kb as well
07:22:39 <joehuang> can Haproxy forward(load balancing) request to kingbird?
07:22:51 <SAshish> that is the problem I suppose
07:22:56 <joehuang> or haproxy drops the packet?
07:23:19 <SAshish> If it forwards and it is discarded at some point I am not sure
07:24:46 <joehuang> for kb, can you try the kb functest bypass the haproxy
07:25:11 <joehuang> from curl http://192.168.0.5:8118/v1.0 first
07:25:26 <SAshish> no 192 series is not accessible
07:25:33 <sorantis> Have you seen this? /etc/apache2/sites-available/25-apache_api_proxy.conf:29:  AllowCONNECT 443 563 5000 6385 8000 8003 8004 8042 8080 8082 8386 8773 8774 8776 8777 9292 9494 9696
07:25:53 <joehuang> no 8118 port allowed!
07:26:19 <joehuang> to Dimitri, this may be the issue
07:27:14 <sorantis> Ashish, try adding 8118 to the list and restart apache. see if it fixes this
07:27:33 <SAshish> it does not
07:27:49 <joehuang> grep 8774 and add 8118 accordding
07:28:07 <SAshish> I feel mapping for 192.168.0.5 => 192.168.0.2 is not happening for 8118
07:29:37 <SAshish> as we are not able to access kb from 192.168.0.2
07:29:42 <sorantis> Iptable /etc/iptables/rules.v4:115:-A INPUT -p tcp -m multiport --ports 9292,9494,9191,8773 -m comment --comment "104 glance" -j ACCEPT
07:29:53 <sorantis> I couldn’t fined an entry for kb
07:30:08 <SAshish> ACCEPT     tcp  --  anywhere             anywhere             multiport ports 8118 /* 106 kingbird */
07:30:19 <SAshish> root@node-4:~# iptables -L | grep 8118
07:30:41 <sorantis> ok, shouldn’t it be in the file?
07:31:35 <sorantis> also this
07:31:41 <sorantis> /etc/haproxy/conf.d/080-glance-api.cfg:3:  bind 172.16.0.3:9292
07:31:49 <sorantis> /etc/haproxy/conf.d/080-glance-api.cfg:4:  bind 192.168.0.2:9292
07:31:50 <sorantis> /etc/haproxy/conf.d/080-glance-api.cfg:10:  server node-4 192.168.0.5:9292  check inter 10s fastinter 2s downinter 3s rise 3 fall 3
07:32:24 <sorantis> i guess this is the mapping
07:32:42 <SAshish> yes
07:32:50 <SAshish> we have file simillary
07:33:34 <sorantis> yes, looking at it
07:34:21 <joehuang> have you restared haproxy after adding the kb configuration?
07:35:04 <SAshish> yes
07:36:42 <sorantis> cat /etc/keystone/default_catalog.templates
07:36:49 <sorantis> weird file
07:38:46 <SAshish> may be it is used during deployment
07:39:02 <SAshish> thats it
07:43:49 <joehuang> ok
07:44:26 <joehuang> this file is used for batch endpont create
07:45:08 <joehuang> can the kb service pingable now from functest container?
07:45:55 <SAshish> no
07:46:44 <SAshish> it is not pingable from controller also on the other internal ip
07:46:46 <joehuang> BTW, the installation and configuration guide has been submitted for review, please have a look, thanks
07:46:57 <SAshish> this is the problem
07:47:49 <SAshish> yeah, will do it. Have you updated comments for the commit done is kb repo as well
07:47:52 <joehuang> we need to have 0.2.0 tag in kingbird after the functest works, and test cases pass
07:48:22 <SAshish> few of my comments are valid on https://review.openstack.org/#/c/349762/ as well
07:48:24 <joehuang> will do that after one is merged, so update once in kb reposity
07:48:39 <SAshish> okay
07:48:44 <joehuang> understand, so review the multisite first
07:48:53 <SAshish> fine
07:49:22 <joehuang> could you add the user guide, you have writen lots of API example, it will be easy for you
07:50:04 <SAshish> yeah, tell me the procedure I will do it
07:50:11 <joehuang> hi dimitri and meiemi, how do you think about it?
07:51:34 <sorantis> about what
07:52:09 <joehuang> tagging and user guide
07:52:47 <joehuang> the colorado branch is planned on Aug 15
07:53:59 <sorantis> I’ve prepared the milestone on the launchpad already. Including the tag that we will use in this release
07:54:00 <joehuang> o, one more, I mentioned in the mail, that some times the on demand sync doesn't sync immediatly
07:54:04 <sorantis> it will 1.0
07:54:19 <sorantis> trying to align with newton release cycle
07:54:32 <joehuang> seems a little early for 1.0 version number
07:54:41 <sorantis> you might have hit the time when on-demand collided with periodic
07:54:45 <sorantis> I don’t think so
07:54:51 <sorantis> we have quota management in place
07:55:03 <SAshish> one usecase we solve then it can be 1.0
07:55:09 <SAshish> the a management
07:55:14 <SAshish> quota management
07:55:22 <sorantis> this could be the first proper release
07:55:32 <joehuang> not reach stable yet
07:55:49 <joehuang> for example, 0.9 also acceptable
07:56:05 <joehuang> beta release
07:56:34 <joehuang> not been tested by user yet
07:57:43 <joehuang> a little worry on the lock
07:58:47 <sorantis> have you checked if you might have hit the time when on-demand collided with the periodic run?
07:58:58 <SAshish> that should not happen
07:59:03 <SAshish> it should not collide
07:59:20 <SAshish> as only in periodic sync I am acquring lock and release it
07:59:23 <joehuang> when kb-engine restart, often can't get the lock
07:59:30 <SAshish> that is because
07:59:34 <joehuang> the message printed
07:59:55 <SAshish> the earlier run has acquired and due to some error it halted
08:00:00 <SAshish> now the lock is there in db
08:00:08 <SAshish> then subsequent runs will fail
08:00:29 <SAshish> before releasing lock, engine is stopped
08:01:07 <SAshish> so next time when we start engine, db has lock acquired earlier, which will not allow to acquire and sync
08:01:09 <joehuang> ok, Ashish, is there any risk for you to submit the userguide and get merged before Aug.15
08:02:28 <joehuang> currently there are quite few projects( in one cycle), so the lock should be released soon
08:02:32 <sorantis> the risk is the immediate problems with testing
08:02:40 <joehuang> but seems it doesn't
08:02:40 <sorantis> you can see that there are issues
08:02:54 <joehuang> test is the first priority
08:03:48 <joehuang> I want to help, but I am not able to access the lab in intel pod
08:04:33 <joehuang> so Ashish and Dimitri fix the test environment issue first
08:04:47 <sorantis> you can help by managing the docs. If the issue persist, Ashish may not have the time to write it
08:05:11 <joehuang> I'll try
08:05:17 <SAshish> I can help with you the API structures and all
08:05:31 <joehuang> ok
08:05:38 <SAshish> thanks
08:05:40 <sorantis> great. I think we can close
08:05:44 <joehuang> #info fix the test issue first
08:05:57 <joehuang> #info joe help to add the userguide
08:06:21 <joehuang> ok, let's close, and hope the test can pass ASAP
08:06:31 <joehuang> #endmeeting