13:01:04 <fdegir> #startmeeting Cross Community CI 13:01:04 <collabot> Meeting started Wed Aug 8 13:01:04 2018 UTC. The chair is fdegir. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:01:04 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic. 13:01:04 <collabot> The meeting name has been set to 'cross_community_ci' 13:01:29 <fdegir> #topic Rollcall 13:01:33 <mbuil> hwoarang: yes! Would it be possible to wait 2-3 weeks? 13:01:39 <hwoarang> #info Markos Chandras 13:01:40 <hwoarang> mbuil: ok 13:01:44 <jmorgan1> #info Jack Morgan 13:01:46 <mbuil> os you want to try new images for the fdgir problem 13:01:49 <mbuil> *or 13:01:54 <mbuil> #info Manuel Buil 13:01:56 <fdegir> here is the agenda on its usual place: https://etherpad.opnfv.org/p/xci-meetings 13:01:58 <hwoarang> i can wait 13:02:24 <fdegir> and the first topic is the issue with the slaves 13:02:32 <fdegir> #topic Issues with Jenkins Slaves 13:02:56 <fdegir> so I went against the motto if it's working don't touch it and broken things 13:03:08 <fdegir> the opnfv vm can't boot 13:03:11 <mbuil> fdegir: I am attending another meeting in parallel, so I might be slow :P 13:03:45 <fdegir> i tried 3 different kernel versions: 4.4.0-112, 4.4.0-122, 4.4.0-131 13:03:50 <electrocucaracha> #info Víctor Morales 13:04:25 <fdegir> one of the 112 and 122 are probably the kernel we used earlier until this morning's update 13:04:38 <fdegir> what else we should look? 13:05:00 <jmorgan1> fdegir: i just deployed yesterday without problem, opnfv host booted 13:05:01 <fdegir> or if one of you want to take a look, the problematic deployment is available on intel-pod16-node3 13:05:12 <fdegir> jmorgan1: is it ubuntu and what's the kernel? 13:05:28 <fdegir> apart from that, when did you do apt update && apt upgrade ? 13:05:47 <jmorgan1> by opnfv host, we mean ubuntu_xci_vm or opnfv vm inside ubuntu_xci_vm 13:05:55 <fdegir> opnfv vm inside ubuntu_xci_vm 13:06:01 <jmorgan1> fdegir: yes, its ubuntu 13:06:12 <jmorgan1> its up and running for me 13:06:19 <jmorgan1> kernel 128 i beleive 13:06:29 <fdegir> ok 13:06:46 <fdegir> can you do apt update && apt upgrade, recreate ubuntu_xci_vm and run xci-deploy.sh again? 13:06:47 <hwoarang> fdegir: does an opensuse vm work/ 13:06:58 <hwoarang> to rule out probs with the VM itself? 13:07:09 <jmorgan1> i had other issues related to k8-calico-onap scenario, but the VM was running 13:07:14 <fdegir> hwoarang: nope 13:07:20 <hwoarang> i mean, if this was triggered by a job, then one of the nodes will have opensuse running 13:07:32 <hwoarang> on the same ubuntu xenial host configuration 13:07:47 <fdegir> hwoarang: https://build.opnfv.org/ci/job/xci-verify-opensuse-deploy-virtual-master/1855/console 13:07:59 <fdegir> hwoarang: https://build.opnfv.org/ci/job/xci-verify-ubuntu-deploy-virtual-master/1854/console 13:08:05 <hwoarang> ok so it's a host thing 13:08:07 <fdegir> same error but it fails faster on opensuse ;) 13:08:17 <hwoarang> no point in updating the images then 13:08:23 <fdegir> i don't think so 13:08:33 <hwoarang> good 13:08:41 <fdegir> is there a way to roll back the complete apt upgrade someway? 13:09:14 <fdegir> or find what kernel version the machine was booted earlier? 13:09:23 <jmorgan1> where are the jenkins slaves? 13:09:24 <hwoarang> journalctl --list-boots ? 13:09:31 <fdegir> cause i did all autoremove to get rid of old kernels as well 13:09:41 <hwoarang> and then look at the logs for the boot you want with journalctl -b<boot id> 13:09:59 <fdegir> there is only 1 boot entry 13:10:02 <hwoarang> :( 13:11:25 <mbuil> fdegir: can you access the VM console? 13:13:38 <fdegir> no 13:14:19 <jmorgan1> does virsh say the VM is paused? 13:14:24 <fdegir> it's running 13:15:02 <jmorgan1> then what is the problen? the vm fails during the boot process? 13:15:17 <hwoarang> but does it matter? it's panicked 13:15:17 <fdegir> http://paste.ubuntu.com/p/5BW7mP3SF5/ 13:15:48 <fdegir> let's move on to the next topics 13:16:09 <fdegir> #topic OSA Shabump 13:16:16 <fdegir> this is problematic as well 13:16:26 <hwoarang> i am looking into that 13:16:31 <hwoarang> we need newer bifrost 13:16:31 <fdegir> ok 13:16:41 <fdegir> i bumped to the latest bifrost 13:16:45 <fdegir> to Manuel's patch 13:16:56 <hwoarang> there is one more commit we are missing 13:17:17 <OPNFV-Gerrit-Bot> Merged pharos: [idf.fuel] Add jumpserver.trunks for mgmt https://gerrit.opnfv.org/gerrit/60743 13:17:47 <mbuil> fdegir: the baremetal patch? 13:18:20 <fdegir> mbuil: i don't remember what was it exactly but after seeing you committed something to bifrost, i tried that one 13:18:29 <fdegir> hwoarang: ok 13:18:33 <hwoarang> no we really need the HEAD of bifrost 13:19:01 <hwoarang> but we will see 13:19:18 <fdegir> ok 13:19:40 <fdegir> the reason why bump is urgent is that I expect the projects will start arriving soon 13:19:45 <fdegir> for masakari and blazar 13:19:58 <fdegir> that's all about shabump 13:20:09 <fdegir> #topic Functest Issues 13:20:15 <fdegir> this is another tricky topic 13:20:28 <fdegir> singlevm test case times out 13:20:32 <hwoarang> that is more urgent i think 13:20:53 <fdegir> i agree 13:21:08 <hwoarang> i had a talk with cedric in the past few weeks and he basically said that nested virt is not proper CI and functest is not designed for such scenarios 13:21:15 <fdegir> well 13:21:20 <hwoarang> so to me the question might be whether we want to keep functest on this level 13:21:23 <fdegir> i probably shouldn't comment on it 13:21:35 <fdegir> what if we were using openstack cloud for patch gating 13:21:38 <hwoarang> maybe we should only use it for smoke + baremetal 13:21:40 <fdegir> getting slaves as vms 13:22:09 <fdegir> and if upstream openstack is doing tempest using virtual machines then what we are doing is not that strange either 13:22:15 <fdegir> but again 13:22:21 <hwoarang> true but we have one more level of virtualization 13:22:33 <fdegir> i think it is same 13:22:50 <fdegir> doesn't openstack ci get slaves from openstack cloud using nodepool? 13:23:01 <jmorgan1> i think so 13:23:03 <hwoarang> yes but we also have clean vm 13:23:09 <fdegir> ok 13:23:18 <fdegir> so openstack gets installed on that nodepool vms directly 13:23:36 <fdegir> then right 13:24:02 <fdegir> i'm wondering how it would perform if we run functest against an xci deployment done directly on host 13:25:15 <fdegir> anyway 13:25:19 <hwoarang> the prob is that if your CI scenario is not considered supported by functets, then we have 0 help from them 13:25:26 <hwoarang> *s/your/our 13:25:37 <hwoarang> so, not sure if we can keep up with that 13:26:29 <fdegir> i really shouldn't say anything 13:26:36 <fdegir> i failed to explain what we are doing and why 13:26:50 <hwoarang> either way 13:26:56 <jmorgan1> we would need one of us to join functest to gain the knowledge 13:27:03 <fdegir> it is not about knowledge 13:27:10 <fdegir> ok 13:27:11 <fdegir> moving on 13:27:43 <fdegir> when we meet next time over a beer, I'll give you details 13:27:49 <hwoarang> we have knowledge to fix stuff. i am submitting patches to fix our cases but i get resistance because of our unsupported case and that's the prob 13:28:03 <hwoarang> wait, because we haven't decided anything and this will block things on SHA bump 13:28:12 <fdegir> nope 13:28:26 <fdegir> we haven't switched back to latest version of functest yet 13:28:32 <fdegir> we are still using the pinned version 13:28:37 <hwoarang> ok then 13:28:38 <hwoarang> :/ 13:28:41 <fdegir> we bump against that 13:28:49 <fdegir> and then come back to uplifting functest 13:28:57 <fdegir> because 13:29:03 <fdegir> anyway 13:29:07 <fdegir> #topic Baremetal Status 13:29:16 <fdegir> mbuil: so, you got it working 13:29:55 <fdegir> mbuil: when do you think we can review the change? 13:31:06 <fdegir> mbuil seems to be focused on his other meeting 13:31:10 <mbuil> fdegir: back 13:31:17 <mbuil> I had to talk in te other one 13:31:26 <fdegir> no multitasking ;( 13:32:11 <mbuil> so, baremetal patch works in ubuntu! I deployed mini flavor several times with SFC- scenario in Ericsson POD2 13:32:36 <mbuil> However, the patch has several hardcoded things and I am currently fixing those 13:32:57 <mbuil> AFter that, I'll split the patch into several patches so that it is easily consumable for reviewers 13:33:04 <fdegir> +1 13:33:20 <mbuil> it would be nice to get a POD where the jumphost uses opensuse 13:33:26 <mbuil> because currently I cannot test it 13:33:51 <fdegir> what do you mean? 13:34:00 <fdegir> why should it matter? 13:34:48 <mbuil> LFPOD4 and Ericsson POD2 have Ubuntu in the jumphost. The code is currently using that information to decide what distro to install in OPNFV and in the blades 13:35:02 <fdegir> mbuil: export DISTRO=opensuse before xci-deploy.sh 13:35:02 <jmorgan1> the jump server is running the opnfv host vm which is deploying to BM nodes 13:35:11 <mbuil> Are you suggesting to define the distro with e.g. a env variable? 13:35:14 <fdegir> yes 13:35:18 <fdegir> that's what we do in ci 13:35:28 <mbuil> oh, that is new to me 13:35:34 <mbuil> then ignore my request 13:35:35 <fdegir> all slaves are ubuntu but we are deciding what distro to bring the nodes up 13:35:57 <fdegir> hwoarang: correct me if I'm wrong 13:36:31 <fdegir> mbuil: we are looking forward to split up patches 13:36:34 <jmorgan1> i'm not sure what we are saying here 13:36:39 <fdegir> thanks for the great work 13:36:55 <jmorgan1> jump server with any OS running the opnfv host VM which deploys to actual BM nodes 13:37:05 <fdegir> jmorgan1: you are not bound by the os of the jumphost 13:37:19 <fdegir> jmorgan1: by default, the host os is chosen as the os for target nodes 13:37:20 <jmorgan1> fdegir: right, that is what I'm saying 13:37:23 <mbuil> jmorgan1: by jumphost I meant the host which hosts the opnfv vm 13:37:37 <fdegir> jmorgan1: but you can change the target node os 13:37:45 <jmorgan1> mbuil: yes, this is what OPNFV defines as jumphost 13:38:20 <mbuil> jmorgan1 the default behaviour is: if jumphost has distro A, opnfv VM gets distro A and compute and controller get distro A 13:38:25 <jmorgan1> agreed, otherwise, how are you going to know what OS to deploy to nodes 13:38:34 <fdegir> by setting DISTRO var 13:38:50 <fdegir> which is what xci actually does behind the scenes if DISTRO is not set 13:39:07 <jmorgan1> ok, i think we are all on the same page 13:39:07 <fdegir> and this was mbuil's question 13:39:27 <jmorgan1> we don't case which OS is on jump host 13:39:39 <jmorgan1> we support threee OS on opnfv host vm 13:39:40 <fdegir> yes, that's it 13:39:51 <jmorgan1> which deploys three OS to BM nodes 13:40:16 <fdegir> anything else mbuil? 13:40:27 <jmorgan1> so mbuil should be able to use opensuse based opnfv VM to test 13:40:50 <mbuil> nothing else. I am pretty busy right now but I hope to give 50% of my time to this 13:41:04 <fdegir> thanks mbuil 13:41:12 <fdegir> #topic k8-calico-onap 13:41:32 <fdegir> jmorgan1: electrocucaracha: how is it going? 13:42:11 <electrocucaracha> fdegir: well, jmorgan1 has been facing the nested virtualization issues 13:42:24 <electrocucaracha> fdegir: trying to test the new scenario 13:42:51 <electrocucaracha> fdegir: we have doubts about the proper syntax for adding a new scenario using gerrit as a source 13:43:32 <electrocucaracha> fdegir: jmorgan1 has a draft for that, we'll need some help on that 13:43:44 <fdegir> electrocucaracha: jmorgan1: https://gerrit.opnfv.org/gerrit/#/c/58945/17/xci/opnfv-scenario-requirements.yml 13:43:59 <fdegir> the version matches to sha of your commit 13:44:14 <fdegir> and the refspec matches to refspec of your change on gerrit 13:44:18 <jmorgan1> we also ran into an issue where we needed to update role (create-vm-nodes) to remove python-libvirt / install virtualbmc 13:45:01 <jmorgan1> i think this issue is resolved 13:45:26 <electrocucaracha> but we didn't find any patch solving that issue 13:45:26 <jmorgan1> fdegir: progress is coming along 13:45:58 <jmorgan1> create-vm-nodes was the 1st problem we had to solve 13:46:23 <fdegir> but now that issue is not there anymore? 13:46:28 <jmorgan1> next we are getting opnfv-scenario-requirements.yml working 13:46:34 <jmorgan1> right 13:46:36 <fdegir> ok 13:46:45 <jmorgan1> might have been fixed upstream 13:47:03 <jmorgan1> when installing virtualbmc playbook, package names changed 13:47:18 <jmorgan1> from libvirt-python to python-libvirt 13:47:49 <electrocucaracha> we tested in ubuntu distro 13:47:51 <jmorgan1> we created a patch to the playbook to remove the old package first, then install new one with different name 13:48:20 <jmorgan1> currently working on opnfv-scenario-requirements.yml 13:49:03 <jmorgan1> ran into nested virt issues then 13:49:18 <jmorgan1> that's all 13:49:28 <fdegir> thanks jmorgan1, electrocucaracha 13:49:40 <fdegir> #topic os-nosdn-osm 13:49:59 <fdegir> the scenario is merged but hit slave issues with the patch integrating the scenario 13:50:12 <fdegir> should be done soon once we get slaves back 13:50:24 <fdegir> #topic os-odl-osm_sfc 13:50:42 <fdegir> i'll try to get this started and then pass it to mbuil 13:51:00 <fdegir> we will also meet with osm guys regarding the next steps 13:51:15 <fdegir> like what should we test and progress with opensuse support 13:51:21 <mbuil> fdegir: this will need to happen in H release probably. No cycles unless somebody helps :P 13:51:25 <fdegir> if anyone is interested joining, ping me 13:51:31 <fdegir> mbuil: that's fine 13:51:47 <fdegir> cause we really don't know how we can test the basic osm yet 13:52:00 <fdegir> so if we can get that pieces for os-nosdn-osm during G-release 13:52:03 <fdegir> i'd be happy 13:52:33 <fdegir> #topic k8-odl-coe 13:52:41 <fdegir> i don't see Peri online so moving on 13:52:53 <fdegir> #topic k8-nosdn-istio 13:53:14 <fdegir> hw_wutianwei_ might be away as well 13:53:46 <fdegir> #topic Testing Objectives 13:53:57 <fdegir> jmorgan1: let's talk about this 13:54:17 <fdegir> for patch verification, we run things in VMs 13:54:36 <fdegir> we do deployment using mini flavor and run old version of functest healthcheck 13:55:12 <fdegir> the reason of running things in VMs (nested virt) is to ensure we always start with clean VM to prevent environment related issues causing unnecessary failures 13:55:20 <fdegir> and utilize slaves more 13:55:36 <fdegir> the next testing is during post merge 13:55:49 <fdegir> the idea is to run functest smoke for merged patches 13:55:55 <jmorgan1> ok, let me explain 13:56:00 <fdegir> and then we take the scenario to baremetal with full functest and yardstick 13:56:06 <fdegir> jmorgan1: ok 13:56:21 <jmorgan1> i think i got my answer wiht our discussion the last two days 13:56:42 <jmorgan1> we have nested virtualization as you pointed out, and baremetal that mbuil is working on 13:56:59 <jmorgan1> no one is doing non-nested VM testing 13:57:25 <jmorgan1> i was curious about what others are doing when we first started getting nested virt errors 13:57:33 <fdegir> others meaning? 13:57:40 <jmorgan1> the xci teeam 13:57:58 <jmorgan1> i think it might be a good use case in the future to have non-nested vm support 13:58:00 <fdegir> i do the same 13:58:07 <fdegir> use ubuntu_xci_vm 13:58:15 <fdegir> but this doesn't mean you have to 13:58:24 <fdegir> you can directly execute xci-deploy.sh 13:58:36 <fdegir> it should be better now given that all the ironic/bifrost stuff is pushed into the opnfv vm 13:59:47 <jmorgan1> yup, i'll look into it when i have time, thanks 13:59:58 <fdegir> ok 14:00:06 <fdegir> #topic Multi-distro Support for New Scenarios 14:00:23 <fdegir> so we want to support all the distros we have 14:00:27 <fdegir> but it is not always possible 14:00:35 <jmorgan1> what is the expectation here? just curious as a new scenario owner 14:00:50 <fdegir> i say start with one distro but develop the scenario in a way that new distro support can easily be introduced 14:01:10 <fdegir> and if you find time, implement support for other distros as well 14:01:41 <jmorgan1> how do we know which distro is supported? been tested? 14:02:04 <fdegir> ubuntu and opensuse are supported for most of the distros 14:02:13 <fdegir> centos is kind of tricky since it is broken upstream osa 14:02:26 <fdegir> so you can start with opensuse or ubuntu 14:03:35 <fdegir> if you are basing your scenario to an existing one, opnfv-scenario-requirements.yml tells you which distros are supported for that scenario 14:04:04 <fdegir> we are over time 14:04:15 <fdegir> #topic XCI nodes in LF Portland lab 14:04:26 <fdegir> jmorgan1: what about these nodes? 14:04:39 <jmorgan1> i added three new systems for xci 14:04:55 <fdegir> it says "LF Portland Lab" 14:04:55 <jmorgan1> i was thinking of using them as develop systems 14:05:03 <fdegir> I suppose it should be Intel Portland Lab, or? 14:05:17 <jmorgan1> but we might want to use them for CI only 14:05:25 <jmorgan1> no, not correct 14:05:40 <jmorgan1> they are in the LF lab, with LF3,4,5 14:05:47 <fdegir> oh 14:05:52 <fdegir> ok 14:05:55 <jmorgan1> thus, LF Portland lab ;) 14:06:18 <fdegir> which pod are these nodes added? 14:06:32 <jmorgan1> no pod, they are their own 14:06:38 <fdegir> ok 14:06:52 <jmorgan1> each systems has one of thesupported OS 14:07:11 <jmorgan1> anyway, lets think about how to use 14:07:18 <fdegir> yes 14:07:24 <jmorgan1> either develop hosts or for CI 14:07:31 <fdegir> we can talk about that separately 14:07:32 <jmorgan1> CI only i mean 14:07:41 <fdegir> we need to go through the resource situation soon anyway 14:07:44 <jmorgan1> i wnated to share that we have them 14:08:33 <fdegir> thanks for that 14:08:47 <fdegir> lets move to the last topic 14:09:03 <fdegir> #topic XCI Developer Guide 14:09:11 <fdegir> jmorgan1: thanks for looking into it 14:09:39 <fdegir> jmorgan1: anything else you want to say? 14:09:48 <jmorgan1> its empty ;) 14:10:04 <jmorgan1> i don't mind starting to work on it, but I'll need some help 14:10:16 <jmorgan1> as the least knowledgable person 14:10:17 <fdegir> jmorgan1: I think Tapio started working on it 14:10:22 <fdegir> can you take a look at it 14:10:24 <jmorgan1> good to know 14:10:28 <fdegir> and perhaps incorporate the comments I posted 14:10:40 <jmorgan1> do you have link? 14:10:51 <fdegir> https://gerrit.opnfv.org/gerrit/#/c/51445/ 14:10:59 <fdegir> please feel free to amend 14:11:04 <jmorgan1> thanks 14:11:27 <fdegir> np 14:11:43 <fdegir> then we end the meeting here if there is nothing more 14:11:50 <fdegir> thank you all for joinin 14:11:53 <hwoarang> thank you too 14:11:58 <jmorgan1> thanks 14:11:59 <fdegir> and welcome back to the ones who had holiday 14:12:10 <fdegir> and have a nice holiday to the ones that are going soon 14:12:14 <fdegir> talk to you in 2 weeks 14:12:16 <fdegir> #endmeeting