16:04:19 #startmeeting JOID weekly 16:04:19 Meeting started Wed May 25 16:04:19 2016 UTC. The chair is arturt. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:04:19 Useful Commands: #action #agreed #help #info #idea #link #topic. 16:04:19 The meeting name has been set to 'joid_weekly' 16:04:32 #info Bryan Sullivan 16:04:39 #info Narinder Gupta 16:04:58 zahra_jahedi: if you are attending meeting please sign here 16:05:03 Agenda item: takeaways from Apex Congress discussion for consideration in JOID 16:06:42 #link https://etherpad.opnfv.org/p/joid 16:07:00 #info Artur Tyloch 16:07:20 #topic Colorado release status 16:07:30 yeah I'm attending 16:08:43 zahra_jahedi: add your name to the meeting notes collected by collabot with this command: #info NAME 16:09:41 #info zahra jahedi 16:09:50 #info David 16:10:59 narinder: we can ear you 16:11:13 David_Orange: but i can not hear any of you 16:11:24 narinder: we ear that :) 16:12:09 David_Orange: any queris on C release 16:12:26 narinder: artur is explaining Converged arch 16:12:31 ok 16:14:01 What is the "converged arch"? Is this documented on the wiki? 16:14:43 bryan_att - not yet 16:34:11 #info prakash 16:46:53 #topic takeaways from Apex Congress discussion for consideration in JOID 16:48:17 shared the status of JOID with Release team stating that os-nosdn-nofeature-ha was using 14.04 currently and other scenerio'sin queue will be with ubuntu 16.04 LTS 16:52:13 #topic OPNFV Summit Berlin 17:02:20 narinder: arturt: and thanks for this work on converged architecture. it seems really interesting, even if i will have a lot of question with my telco background :) 20:13:27 narinder: ping 20:13:38 fdegir: pong 20:13:46 fdegir: how are you doing today> 20:13:52 narinder: good 20:13:55 narinder: how are you? 20:14:15 fdegir: i am good thanks in Austin at Big communication event. 20:14:40 narinder: good for you 20:14:43 narinder: now 20:15:02 narinder: do you remember the mail I sent a week or two ago 20:15:05 fdegir: yeah i am out and working on few stuff 20:15:15 fdegir: regarding? 20:15:15 narinder: regarding changing how we run jobs on jenkins 20:15:35 narinder: you know, intel-pod5 runs brahmaputra jobs, intel-pod6 runs master for joid 20:15:40 narinder: we are changing it now 20:15:53 narinder: and your jobs will be regenerated 20:15:54 fdegir: ok 20:16:07 narinder: like joid-os-nosdn-nofeature-ha-baremetal-master and brahmaputra 20:16:18 that sounds good to me and i am working on config not part of joid 20:16:36 meaning that we will run any joid job on any pod joid has 20:16:36 so that it will input to joid and you can deploy the stack dynamically 20:16:44 that's fine 20:16:51 woe thats fantasic totally 20:16:54 but the jobs will roam on 2 pods randomly 20:17:06 okey only two pods :) 20:17:48 thats fine but can i define to have some way to not to reschedule it on a regular interval initally 20:17:49 narinder: this is the first step 20:17:53 fdegir: ok 20:17:54 narinder: later - perhaps d release 20:18:01 gotch you 20:18:02 narinder: we want to put all pods to common pool 20:18:05 anyway 20:18:08 I'll send a new patch 20:18:09 i am totoally +1 to it 20:18:11 and explain details there 20:18:19 sure sounds good to me 20:18:27 so please read the details and put your objections there in case if you have any 20:26:02 fdegir: sure 04:25:33 narinder: when you get back on, the charm and result from trying to deploy it are attached to the page https://wiki.opnfv.org/display/copper/Installer+Support 14:44:10 narinder: do you or your charm experts have some time today to help me move forward on the Congress charm? 14:45:23 bryan_att: yes i can sit with you after 2:00 PM CST 14:45:55 bryan_att: and if needed then we can ask queries in #juju channel 14:47:25 narinder: OK, thanks 2PM is great. The first issue I have is the error line "ERROR juju.state.unit unit.go:738 unit congress/0 cannot get assigned machine: unit "congress/0" is not assigned to a machine". Will the juju channel guys be able to help me with that? 14:48:28 bryan_att: yes you can point towards your charm code and they can. 14:48:53 bryan_att: can u run juju status and send me output? 14:49:22 and how did you deploy? 14:49:41 narinder: here it is https://www.irccloud.com/pastebin/WUHmkYZE/ 14:50:42 narinder: log is at https://wiki.opnfv.org/download/attachments/6821110/congress-charm.log?version=1&modificationDate=1464236675441&api=v2 14:53:20 ok error you do not have enough hardware machine in maas to deploy congress on physical machine 14:53:44 so either you have to use container or colocate the service. 14:54:27 since you are using the maas provider so any deployment looks for a machine from maas 14:54:32 narinder: ok, is there something I can copy as container config for this? Really I want to deploy it on the same container as Horizon. 14:55:05 to colocate the service you can do juju deploy loca:trusty/congress --to 1 14:55:38 narinder: I just walked thru the guide provided without really understanding it, so I'm sure there are things missing. 14:56:36 narinder: is the command "juju deploy local:trusty/congress --to 1" ? The command I used was "juju deploy local:liberty/congress" 14:56:55 sorry it could be liberty 14:57:07 path depends upon where your local charm is 14:57:12 what is the "--to 1" ? 14:57:31 --to 1 means deploy service to machine 1 in MAAS 14:58:03 in this case it is 1 started 1.25.5 node1-control.maas /MAAS/api/1.0/nodes/node-dc124fbe-222b-11e6-a6b6-525400525c72/ trusty arch=amd64 cpu-cores=4 mem=16384M tags=control availability-zone=default 14:58:15 ok, thanks. I can try that. How would I direct it to deploy on the same containter as Horizon? 14:58:17 which is node1-control.maas 14:58:52 for that you need to say --tp 1/lxc/6 14:59:06 --to 1/lxc/6 15:00:29 ok, let me try that. I don't want to bug you too much, so I'll followup on juju if that's the right place. 15:00:58 you can ask me question and also on #juju channel they will be supportive 15:01:55 ok, first is I just tried that and got "ERROR cannot add service "congress": service already exists" - how do I remove it to try again? 15:03:22 maybe via "juju remove-service"? 15:04:42 narinder: I got this error trying again https://www.irccloud.com/pastebin/RuSAEWdG/ 15:07:32 narinder: same error with "juju deploy local:liberty/congress --to 1" 15:09:14 afk for a bit 15:50:41 narinder: I'm going to ask for followup on juju. I've documented the current status at https://etherpad.opnfv.org/p/copper 16:07:12 narinder: I changed the command to "juju deploy local:trusty/congress --to 1" and it seems to be getting further. The instructions caused me to think I should name the series based upon OpenStack release but apparently that's not how you named it for JOID. 16:08:12 narinder: hi 16:34:14 narinder: hook failed: "install" log lines for "juju deploy local:trusty/congress --to 1" https://www.irccloud.com/pastebin/Hn5V1UgP/ 16:41:01 narinder: orange pod2 is down -> a big auth issue from nova with bad tokens 16:42:10 narinder: i found something more than stange, on all clusters, all containers have the vip, making pacemaker restart solve the multi vip issue, but it seems not normal 16:43:44 bryan_att: unit-congress-3[1725099]: 2016-05-26 15:58:25 INFO unit.congress/3.install logger.go:40 TabError: inconsistent use of tabs and spaces in indentation 16:43:53 which is the root cause 16:44:41 David_Orange: vip should be configured in all right to get it services. 16:44:51 may be cluster type is active-active 16:45:58 this is not the case on pod1 16:46:09 only one vip for 3 servers 16:46:32 and restarting pacemaker restore 1 vip for 3 servers 16:47:16 or it is a new behavior 16:48:30 but after a moment, vip is back on each server 16:48:55 it may not be linked but this is different between pod1 and pod2 16:53:29 David_Orange: i need to understand the changes in pacemaker before i can comment further. 16:53:40 ok 16:55:23 it is the same for all clusters of pod2 16:55:54 i stopped 2 pacemakers on keystone services, to check if auth problem is solved or not 16:58:23 David_Orange: ok is it resolving? 16:58:56 not for the moment 17:02:45 still the same issue: 17:02:47 (keystone.middleware.core): 2016-05-26 17:02:21,788 WARNING RBAC: Invalid token 17:02:47 (keystone.common.wsgi): 2016-05-26 17:02:21,789 WARNING The request you have made requires authentication. 17:03:37 i have to go, i will follow debug tomorrow if you don't have time for it 22:01:05 narinder: quick question if you are around. How can I get the values typically found in the admin-openrc.sh without having to download it? Are there juju commands I can use for that? 22:02:32 bryan_att: yes you can use juju get to get the charm configuration where value resides 22:03:34 narinder: ok, do you have a link to where that is described? I see some related guidance at https://help.ubuntu.com/lts/clouddocs/en/Installing-OpenStack.html 22:04:19 bryan_att: i think you are loking for vip for keystone service and also regiona name and username name and password 22:04:38 yes 22:05:51 OS_AUTH_URL, OS_TENANT_ID, OS_TENANT_NAME, OS_PROJECT_NAME, OS_USERNAME, OS_PASSWORD, OS_REGION_NAME 22:06:11 thats all I need to use the OpenStack CLI commands in the install scripts 22:06:55 usually at the end we create it and placed under ci/cloud directory 22:07:25 and script to create is in ./nosdn/juju-deployer/scripts 22:10:08 and this scripts uses JUJU to get the ip of keystone and get it embedded in to the adminrc file 22:10:21 script name is openstack.sh 22:11:16 unfortunately both intel pod5 and pod6 have started a fresh deployment so can not give you exact commands 22:13:00 ok, np, I have the file. It looks like the value are hard-coded. Is it acceptable for me to just call that script as part of the install process (bash-based)? 22:13:21 but you need to change the networking 22:14:06 part as it download the image and upload into cloud and also try to create the network also 22:15:01 but if you are using deploy.sh creating adminrc already happening 22:15:20 I can get the keystone IP address OK. I just needed to know what the other values are. Looks like they are all hard-coded. I can just use the same values, but my script will break if they ever change... but anyway we are moving to charm. I just wanted to simplify the bash install process. So I think I am OK with hard-coding them. 22:15:35 where is the adminrc that is created? 22:15:55 under joid/ci/cloud/ 22:16:33 ok, I see it now. I'll just call that in the script. Assumes I know where it is. 22:16:47 ok 22:17:45 Do you have a standard practice for where you put the joid repo, e.g. at ~/joid ? I have been putting it at ~/git/joid 22:18:00 (under the ubuntu user) 22:18:15 just want to make sure I reference it correctly 22:18:49 if you are using jenkins then you can find openrc file under ~/joid_config 22:19:29 otherwise it can go anywhere based on user choice but from ci directory it is under cloud directory always 22:20:51 ok, that's a little tricky, not reliably the same folder. So for now I will hard-code the values except for the keystone IP. Simpler. 22:21:14 bryan_att: i am sure we have juju way to find it 22:21:59 as we can use juju get keystone and look for all config options and then decide dynamically 23:16:36 narinder: if you 23:17:51 narinder: are there... I need to access mysql on the horizon lxc but I'm getting "ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)" - previously I didn't get this error. Is there a password I need to use? 23:20:52 bryan_att: can we pose this query on #juju 23:20:53 ? 23:21:20 sure, I just thought this was an OPNFV-specific thing 23:22:00 bryan_att: no i am using generic mysql server and i am sure there willl be beter way to connect or run the query 23:22:41 narinder: I'm trying to create the database for Congress 23:23:22 but I can work around this I think by creating my own LXC for Congress, for now 23:23:23 bryan_att: yeah for that you do not need to connect. I think you can send your sql command to mysql charm and they will take care of creating it 23:23:42 narinder: yes, but I'm not there with the charm yet... 23:24:18 bryan_att: we use percona-cluster and password set in bundle i think 23:24:30 which bundle you deployed? 23:24:58 narinder: not sure what you mean... this is a plain JOID install for liberty 23:25:12 with odl or nosdn 23:25:13 ? 23:25:19 nosdn 23:25:19 ans also ha or nonha? 23:25:24 non-ha 23:26:09 ok i do not set it in bundle for non ha 23:26:19 but for now you can set it using juju 23:26:30 let me give you command and then you can connect 23:28:21 juju set mysql/0 root-password=anypassword 23:28:33 then you can use anypassword to access the mysql 23:28:58 if you need to sey sst-password 23:29:21 then juju set mysql/0 sst-password=my-sst-password 23:29:44 "ERROR "mysql/0" is not a valid service name" 23:29:55 may be you have different number 23:30:19 can you send me juju status --format tabular 23:30:37 also try replacing mysql/0 to mysql 23:30:47 https://www.irccloud.com/pastebin/zHexsNsB/ 23:31:19 that worked (just mysql) 23:32:03 ok yeah it will change it to all instances running my bad 23:32:16 it should be mysql only then 23:32:37 now you can try to connect to mysql using this root password 23:32:52 but looks like it didn't work - I still get denied even when I set it to "nopassword" 23:33:04 just wait 23:33:18 chec juju status mysql/0 23:33:26 and see what the status says 23:33:53 as it take time all other services has to repolicate and restart to adopt new password 23:34:07 juju status --format=tabular should not show execting 23:39:30 ok, I didn't see it change. let me try creating a new LXC and see if that works. 00:37:06 ok 01:50:29 fdegir: how do i enable build job for intel pod6 which is my dev server 01:50:35 sorry dev pod 13:24:25 David_Orange: hi good morning 13:24:34 Hi 13:25:17 David_Orange: i will be checking in the code for DPDK today. 13:25:31 great 13:25:34 David_Orange: and next week for LX| enable_dhcp | False | 13:25:34 D 13:25:43 next week for LXD 13:25:55 good news 13:26:23 i debug pod2 yesterday but had to reinstall it on stable today 13:26:26 i know we still has to work out issues with ODL though 13:26:39 is stable working well? 13:26:49 yes, it blocks me on public api 13:26:54 it seems 13:26:58 ok 13:27:17 so we need to see the difference i know one change in SR2 od ODL with latest 13:27:33 and need to fogure it out other thing else well in terms of bundle 13:27:36 we will see next week, ixia is coming here in France to install ixia virtual loader 13:27:44 ok 13:27:55 ok 13:28:15 can u pass me bundles.yaml from ci ? so i can compare? 13:30:02 i still feel it is SR1 vs SR2 where ODL SR1 works and SR2 has issues 13:32:28 you need bundles from pod1 ? 13:32:34 i had issues with both 13:32:39 odl bundle where it works 13:32:47 oh 13:32:59 that means stable is not working? 13:33:12 first with SR1 then i upgrade to SR2 13:33:38 with master branch is it? what about stable branch? 13:33:44 no, i think i used master bundle template before your switch to SR2 13:33:54 i use master on pod1 13:34:00 stable on pod2 13:34:00 ok 13:34:12 and both pod have issues is it? 13:34:22 only pod1 13:34:37 on pod2 it was another issue 13:34:43 ok so i need bundles.yaml from pod2 stable 13:34:56 pod2 is nosdn 13:34:59 sorry 13:35:02 oh ok 13:35:20 nosdn works with master as well 13:35:29 ok nosdn is not a problem 13:35:34 looking for odl 13:36:03 the only i have is pod1 and i modified it for public api 13:36:47 ok 13:37:01 no you can keep working on public api 13:37:11 i will see if i can reuser intel pods 13:38:24 btw i have changed the node naming concention in maas as well to match the physical location 13:40:04 ok 17:35:42 narinder: any idea on the issue I reported on #juju? gnuoy seems to be OOO 17:37:50 narinder: how do I stop a juju deploy in progress? This having to wait for the deploy to reach the next "failed" state then "resolved" etc step by step thru the hopeless process is pretty time consuming. How can I abort a deploy and remove the service so I can start again? 17:38:28 narindergupta: ping 19:35:13 bryan_att: will you please tell me issue again sorry i was in meeting? 20:35:41 narinder: how do I stop a juju deploy in progress? This having to wait for the deploy to reach the next "failed" state then "resolved" etc step by step thru the hopeless process is pretty time consuming. How can I abort a deploy and remove the service so I can start again? 20:36:08 narinder: also any idea on the issue I reported on #juju? gnuoy seems to be OOO 20:37:13 narinder: related to these charm-congress deploy errors https://www.irccloud.com/pastebin/ozjLJ7Kw/ 20:37:15 bryan_att: you can always terminate it using juju destroy-service and then run terminate-machine 20:38:06 bryan_att: in this looks like few packages are not there? 20:38:24 narinder: destroy-service doesn't seem to work when the service is in hook-failed state 20:38:45 bryan_att: then terminate-machine then the lxc name should do 20:39:14 bryan_att: specially --force 20:39:16 narinder: re the deploy, yes gnuoy claimed ability to deploy and posted a log. But when I try exactly the same sequence I get that error 20:39:35 and you are tring the liberty 20:39:56 narinder: yes, liberty 20:39:58 can u run sudo apt-cache policy on congress-server 20:40:18 sorry command is sudo apt-cache policy congress-server 20:41:33 N: Unable to locate package congress-server 20:41:53 that is problem 20:42:11 can u send me out of la /etc/apt/source.list.d/ 20:42:17 but gnuoy was able to deploy without that error. How? 20:42:43 ls: cannot access /etc/apt/source.list.d/: No such file or directory 20:42:45 lets check. He may have the liberty repos added 20:43:10 sorry it is sources.list.d 20:43:13 not source 20:43:39 cloudarchive-liberty.list should be listed 20:44:05 yes it's there 20:44:13 nope it is not in cloudsrchive-libert 20:44:25 do you know where can we find this package? 20:44:33 https://www.irccloud.com/pastebin/1iX1irSx/ 20:45:01 how do I get the machine-id for the terminate command? 20:45:07 not on jumpshot on the system where install issue is there 20:45:33 juju status --format=tabular should show the machine id 20:45:38 if it is container 20:45:50 then it might be 1/lxc/0 20:46:02 or something liske that against service 20:46:34 looks like congress-server package is not part of liberty repo in ubuntu 20:46:46 then how did it work for gnuoy 20:47:05 here is the status output - what is the machine ID for congress? 20:47:07 https://www.irccloud.com/pastebin/ENy41si5/ 20:47:08 it must be in ppa some where 20:47:37 bryan_att: i need output of tabular format 20:47:47 as short only talks about services 20:47:57 https://www.irccloud.com/pastebin/zl7zO2UH/ 20:48:38 thats bad looks like you are trying on bare metal itself rather than container 20:48:52 so it will be difficult to terminate the machine' 20:49:01 well it's per the charm-congress, right? 20:49:05 as other services are running 20:49:20 yes by default with maas provider it is bare metal machine 20:49:33 The command I used was "juju deploy local:trusty/congress --to 1" 20:49:43 that is for bare metal 20:49:55 what should I use for a container? 20:50:53 in bundle we use lxc:nodes=0 20:50:59 I guess I could use the Horizon container ID? But what if I wanted a new container for Congress? 20:51:33 or lxc:nodes=1 depend upon which machine you want 20:51:44 i need to check correct command 20:51:58 but you can try --to lxc:nodes=0 20:52:20 what does "nodes=0" mean? 20:52:48 i deployed ubuntu services and give name as nodes 20:52:56 and uses two macines 20:53:30 nodes=0 means deploy me my server in lxc container on the machine with instance id 0 and service nodes is deployed 20:54:14 ok, once I get this stopped and removed I will try again with that 20:54:32 but I need to resolve the package issue firsrt 20:57:27 I guess everyone is bugging out for the holiday weekend. Understandable, I just want to continue making progress... anyway I'll try again once I get it cleaned up. If you find out anything about the packages or how gnuoy overcam that, let me know if you can. 20:57:40 gotta go get my lunch out of the oven 20:58:13 gnuoy is based in UK so he will be off by now 20:58:28 bryan_att: ok but let me check the congress package first 20:59:54 narinder: sure, np and thanks for your help 21:00:08 bryan_att: i figure it out the congress package issue 21:00:34 bryan_att: it seems congress is enabled with mitaka only on xenial not with trusty 21:00:48 bryan_att: check this https://launchpad.net/ubuntu/+source/congress/3.0.0+dfsg1-1 21:01:07 it builds only for xenial 21:02:04 bryan_att: so only option for you to use is mitaka on xenial 21:03:09 which you can deploy it with joid master branch ./deploy.sh -o mitaka -s nosdn -t nonha -d xenial -l 21:03:45 narinder: do I have to have xenial installed on the jumphost? 21:04:07 no not neede 21:04:11 d 21:04:17 you can deploy using current tool 21:04:35 narinder: ok, I will try that. 21:04:48 as maas is ready to deploy the xenial as well now. When did you isntall MAAS? 21:05:51 a couple of days ago at least 21:06:02 I can reinstall maas if needed. 21:06:16 no need to reinstall MAAS 21:06:25 just login into maas page and click on images 21:06:47 check whether it has image downloaded for both 14.04 and 16.04? 21:07:07 if it does then do not do anything 21:07:21 otherwise check the 16.04 and click on apply 21:07:25 16.04lts is there 21:07:37 and images how many images are there? 21:08:11 do you mean buttons I can select or something in a list? 21:08:51 unders the button selection do you see 16.04 and 14.04 both or only 14.04? 21:09:08 there is a 16.04 button 21:09:14 is it checked? 21:09:31 I just checked it and hit apply 21:09:49 it says "Step 1/2: Region importing ." 21:09:59 ok sounds good lets wait for completion 21:10:02 then clean it 21:10:15 and run deploy according to my instructions above 21:10:32 ok, will do. fingers crossed! 21:10:44 i know as this is mitaka with xenial 21:11:05 i know mitaka works with 14.04 fine 21:11:17 but looks like congress package is not there in trust 15:53:09 narinder: hi 15:53:26 David_Orange: hi David how are you? 15:53:32 narinder: fine and you ? 15:53:46 David_Orange: yeah i am doing good thanks 15:54:02 i worked today on public api 15:54:49 i set a script to create a reverse proxy on jumphost for CI/Tests only (where we can not set NAT on the default router) 15:55:24 it seems to works well on pod1 but now i wish i can try to run functest 15:56:30 but first i have to reinstall pod1 and be sure i wont have the "No valid host was found" error. Did you progress on that N 15:56:31 ? 15:59:10 narinder: i also write an how-to: https://wiki.opnfv.org/display/joid/How+To#HowTo-ConfigureaccesstoPublicAPIonHAmode 16:02:02 David_Orange: please try nosdn that should work 16:02:15 it seems odl issue is more complicated 16:02:22 and i am not getting much help on that 16:02:41 ok, i will do that tomorrow 16:02:57 and let you know if functest pass 16:09:59 sure that will be great 16:10:37 also i am changing few position of variables etc in yaml files to include more support of lxd and other storage options. 16:11:15 so in worst case you might need to resumbit the patch after rebasing i am hoping it won't conflict 19:12:51 narinder: ping 19:12:55 narinder: regarding pod6 19:13:01 narinder: you said it is your dev pod 19:13:10 narinder: but it is a CI pod 19:13:12 fdegir: hi 19:13:38 fdegir: i use it for both and once dev is done then it will be used for CI 19:13:48 ok 19:13:51 otherwise i will left with no dev pod 19:13:58 but since we switched using labels 19:14:08 jobs might start on both pods 19:14:34 yeah i see that and my dev delpoyment usually cleaned so that may be reason 19:26:58 well, you need to bring this to infra wg 19:33:00 fd ok 15:26:31 David_Orange: thanks for correcting the password. Can u try to deploy it and see whether it works or not? 15:27:00 narinder: also i have implemented to use pass mutiple comma seperated feature during deployment 15:27:02 narinder: hi, deploy is actually in progress with my publicapi changes 15:27:37 ok cool 15:27:44 narinder: ok great 15:28:34 David_Orange: further i will work on removing the node specific data like putting interface name in ext-port i am planning to replace with MA 15:28:38 C address 15:29:06 and also once maas-deployer will be update then i can remove the auto enabling of interface data in joid 15:30:06 narinder: once i finish that i will test dpdk, but i think i will also contribute to you new deploy script (add password management, pep8 syntax...) and also think of how to merge multiple sdn controller template, but it will be later 15:30:47 David_Orange: thanks for that David and really appreciated the work you are doing 15:30:53 with joid 15:32:04 thanks narinder, the transatlantic sync is not always easy, but i like to contribute on joid :) 15:32:22 :) 16:03:42 arturt: Error: Can't start another meeting, one is in progress. Use #endmeeting first. 16:03:50 #endmeeting