13:58:19 <[1]JonasB> #startmeeting weekly Fuel@OPNFV sync 13:58:19 Meeting started Thu Sep 17 13:58:19 2015 UTC. The chair is [1]JonasB. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:58:19 Useful Commands: #action #agreed #help #info #idea #link #topic. 13:58:19 The meeting name has been set to 'weekly_fuel_opnfv_sync' 13:58:28 <[1]JonasB> #info Jonas Bjurel 13:58:38 #info Stefan Berg 13:58:55 #info Michal Skalski 14:01:29 #info Szilard Cserey 14:01:40 Hi Everybody :) 14:01:48 <[1]JonasB> Hi Michal and Szilard 14:01:49 Hey Szi! :) 14:02:05 Hi Stefan! 14:02:07 <[1]JonasB> Mr berg - welcome 14:02:12 Hi Jonas 14:02:17 Mr Bjurel, a pleasure. 14:02:19 Hi Michal! 14:02:28 Hi, 14:02:34 Hi Ruijing! 14:02:34 Hi 14:02:46 <[1]JonasB> Ok lets focus todays meeting on SR1 14:02:52 Hi, szilard, Michal 14:02:54 ruijing: ni hao 14:03:07 <[1]JonasB> #info Meeting focus is on SR1 14:03:36 <[1]JonasB> #topic SR1 14:03:51 #info Daniel Smith 14:03:53 <[1]JonasB> Lets start with building/issues 14:03:53 hello all 14:03:55 Hi Daniel 14:03:57 <[1]JonasB> Hi Dan 14:04:17 <[1]JonasB> #info Build issues: 14:04:21 <[1]JonasB> I can start 14:05:18 <[1]JonasB> #info There is a problem with the build cache archive wich sometimes has trailing garbage in it 14:05:59 <[1]JonasB> #info I'v seen that the garbage comes in when constructing the archive with tar -caf... 14:06:38 <[1]JonasB> #info I have a patch which will strip the trailing garbage, but I have not been able to find out the root cause. 14:06:55 <[1]JonasB> Thats what Im working on - comments? 14:07:25 [1]JonasB: how much time it adds if we disable the caching? 14:07:37 <[1]JonasB> 1 hour per build 14:07:38 meaning; how crucial is the caching? 14:07:47 ok 14:07:54 <[1]JonasB> I think we need it 14:08:02 I agree 14:08:03 i think that for auto builds, not such a big deal 14:08:09 but for people doing work you wanna have that there 14:08:14 cause the iteration times are really long 14:08:19 and 1 bug means ya start over :) 14:08:43 So Jonas, this is occuring also on local builds? 14:08:45 so we can enable cache in default 14:08:57 we can perhaps disable it for daily runs until sr1 is out 14:08:57 and disable cache in jenkins 14:09:07 That's to say, it is not environment dependent? 14:09:23 <[1]JonasB> s_berg Have not seen that on my machine, but I will try some more 14:10:06 @Fatih - that was my thinking for the CI stuff / auto builds 14:10:06 lmcdasm: Error: "Fatih" is not a valid command. 14:10:11 if you want, I can setup a job and unpack the cache many no of times a day to see what happens 14:10:12 Fatih - that was my thinking for the CI stuff / auto builds 14:10:16 dont use cache 14:10:18 [1]JonasB: That's my point, I'm not seeing it either... 14:10:21 and then people have the option locally if they want 14:10:28 (i dont have an issue either on my build system - just FYI ) 14:10:29 <[1]JonasB> We already to day see the CI_pipeline quing up commits due to lack of resources, and it would get a lot worse if we disabled the cache 14:10:39 really? 14:10:40 lmcdasm: for verify, cache is needed 14:10:45 for dailies, it can be disabled 14:10:46 ahh - for verify 14:10:47 damn 14:11:06 i was thinking only of the daily / nightly builds 14:11:13 Jonas - do we want to prop up another build server? 14:11:18 we have enough resources 14:11:23 <[1]JonasB> But my patch should fix it, the thing is that I dont understand what is going on 14:11:30 hee hee 14:11:57 <[1]JonasB> Ok lets move forward to the other build issu: s_berg 14:12:02 jsut one sec 14:12:09 Fatih - do you want another Jenkins server setup? 14:12:09 I'm a build issue now? Wow! :) 14:12:11 would that help? 14:12:18 So. 14:12:24 I'm setting a job now 14:12:39 which will unzip/untar the cache every 3 minutes or so 14:12:43 I've got a nice little build setup which creates one data container and one Apache container. 14:12:48 on current build server 14:12:52 to see how frequently it happens 14:12:59 Moved the fuel-createmirror onto it, and it's a breeze to rsync over the needed repos. 14:13:04 will report back and then lmcdasm we can talk about a new build server 14:13:07 nice one Sberg! 14:13:20 The thing now is to settle on where we want to go with this. 14:13:38 <[1]JonasB> fdegir: The problem isnt the decompression/un archiving. It is when the compressed archive is created. 14:13:40 1. Just have a tar.gz, and untar that into the right place onto a deployed Fuel node. 14:13:49 [1]JonasB: ok 14:13:54 2. Add it to the Fuel ISO directly when building. 14:14:06 suggestion Sberg, that we deliver as a tar with a script that props up your docker container 14:14:07 then I withdraw my suggestion 14:14:11 3. Package up the containers so they can be deployed anywhere. 14:14:19 and then call this script with a post-install in Fuel? 14:14:20 4. Make a plugin to be loaded onto Fuel post-deploy. 14:14:31 s_berg: fuel-createmirror probably not download java package and its dependencies which are required for ODL 14:14:39 <[1]JonasB> s_berg: I have a little test script that tars and untars. Could try to run that on the build server. 14:14:54 mskalski - would we not have the java delivered with the ODL plugin? 14:15:01 rather than seperately? 14:15:10 <[1]JonasB> On thing at the time guys. 14:15:28 <[1]JonasB> Let s_berg info status on the other build system 14:16:22 #info A procedure to populate a Docker data container with needed Fuel repositories is in place, together with building another container running Apache and serving said repos. 14:16:44 #info What's not settled is how to package this up. 14:16:58 arno.iso doesn't include ODL plugin rpm, right? 14:17:09 good question :) 14:17:19 would be nice to contain it 14:17:38 <[1]JonasB> I would vote for a plugin, or do we get catch22? 14:17:40 well - adding it into the ISO is not an issue if the package exists in the fuel build substructure 14:17:47 s_berg: plugin options probably will not work 14:17:52 i dont think it will work 14:18:15 either - hence saying that a RPM and some script post install onto the FUEL server it self (since plugin goes to the infarstructure - not the FUEL node as i understand it) 14:18:22 yes. we need to include ODL plugin rpm and auto install in master. 14:18:23 s_berg: pre_deployment task of the plugin is run after image of system is builded 14:18:43 <[1]JonasB> Ok a question, would it work to have it as a plugin when we come to Fuel 8? 14:18:46 Autodeployer will locate the plugin and do the installation 14:19:07 lmcdasm: there is a possibility to run task on Fuel master from plugin but it will be to late 14:19:18 My priority list for this: 1. Tweak the repos into the ISO during build for now, 2. Let the deployer untar a repo structure and 3. Package up this as containers and figure out a way to deploy them. 14:19:27 i think Michal and I are saying the same things.. you will need to prop up this container as soon as FUEL VM is installed - i know you hate it Sberg, but pre-deploy would be great for this (and could se the value for the mirror in FUEL Env settings page) 14:19:44 mskalski: OK, plugin off the table for SR1 - good to narrow choices! 14:19:49 agreed - plugin is too late int the chain 14:19:56 s_berg: nice 14:20:26 lmcdasm: Why would we need pre-deploy? Why not simply prop it onto the ISO we build in that case? 14:20:43 yes. if too late, we can include it at least 14:20:48 well - i was thinking that with pre-deploy then we remove the human part of changing the Settigns in ENv creation to point to the new URL 14:20:53 <[1]JonasB> I would vote for having it in the iso for SR1, and try to get it as a pre-deploy plugin for B-release 14:21:14 yes. 14:21:17 since if i get it now, we would do the following: a) isntall FUEL b) login and prop up container c) define environemnt d) run plugins e) change settings f) deploy 14:21:22 [1]JonasB: +! 14:21:24 +1 14:21:45 i like the evolution idea - lets get the artifacts onto the ISO and in there (so we can use em manually) for SR1 a 14:21:51 and then work towards automation of it for Brel? 14:21:53 +1 14:21:57 pre-deploy is to late to provide package for build system image: provision node(instal system) -> pre-deploy -> deploy -> post-deploy 14:21:58 +1 14:21:59 Brel? :) 14:22:04 B-release 14:22:07 I like it! 14:22:09 sorry berg - im horrible today 14:22:16 Sounds like a river. :) 14:22:21 :D 14:22:42 OK, seriously, I think that's the quickest fix for SR1 - we put 'em in there. 14:23:00 again - we wouldnt "need" predeploy (just to finish) - but I can see people making mistakes and we can (if we want) help em 14:23:00 The we evolve. But pleeeease, let the pre-deployer remain buried.... 14:23:03 <[1]JonasB> So can we agree on having it in the iso for SR1 and come up with something better for B-river 14:23:07 yes yes - 14:23:22 get the artifacst in there and maybe I can do some work (i know - imagine) on seeing how to automate it 14:23:32 yes and autodeploy will take care the rest of it 14:23:34 maybe something simple like a /etc/init.d/SXX script 14:23:41 (or have Szi just do it for autodeploy) 14:23:42 hahaha 14:23:47 yes. maybe we just create JIRA and include in release nodes 14:23:49 Yes, we're just changing the config part to points to the Fuel IP. 14:24:03 well.. since its local 14:24:10 cant you jsut use localhost 14:24:11 So the rest is just a dea config, nothing to automate other than making that change. 14:24:13 that way its portable? 14:24:55 I assume (Michal?) that the URI must resolve also when used from computes/controllers. 14:25:02 <[1]JonasB> #Agree The local repo mirror will in SR1 be in the iso and will be running in a container on the fuel master. For B-release we will do something smarter not requiering a OPNFV propriatary .iso 14:25:07 So localhost is probably not going to fly. 14:25:19 s_berg: yes 14:25:23 S_berg - ahh.. i thought it was FUEL gets mirror and then distributes - ok then 14:25:24 No! 14:25:29 Not in a container. 14:25:29 bah 14:25:31 haha 14:25:43 <[1]JonasB> ahh on bare metal 14:25:47 <[1]JonasB> #undo 14:25:47 Removing item from minutes: 14:25:55 (I hope not at least). Yes, we just pre-populate /var/www/nailgun. 14:26:09 <[1]JonasB> #Agree The local repo mirror will in SR1 be in the iso and will be running on the fuel master. For B-release we will do something smarter not requiering a OPNFV propriatary .iso 14:26:14 question 14:26:25 <[1]JonasB> Yes mister smith 14:26:38 do we want to make this mirror fetching static? Meaning that when we cut SR1 - the mirror repo is frozen in the container (or in git somewhere) 14:26:57 so that we have a solid REPO that is always used - or do we keep it dynamic (meaning that from install to install you might have repo updates) 14:26:58 ? 14:26:58 but way you want to build another container for repo? I think we can just place packages in an appropriate place on fuel master 14:27:16 It would probably make sense, from a build time perspective, to make the repo tar a separate internal artifact. It takes like 85 minutes to create it... 14:27:18 <[1]JonasB> lmcdasm: Yes 14:27:26 (my question michal - you're too smart) cause i was going to say if its a static repo, then just populate /var/ww/nailgun once and leave it there 14:27:52 mskalski: Yes, agree to that, just a matter of putting it in the right place. 14:28:39 <[1]JonasB> Ok I think we have a plan 14:28:42 I'll start fiddling with that tomorrow, hopefully not rocket science. 14:29:05 s_berg: ping me if you need help 14:29:12 mskalski: Thanks, will do! 14:29:26 <[1]JonasB> Next s_berg: What about the build issue creating files with root as user 14:29:35 question: all rpm packages are built by fuel jenkins or upstream jenkins. 14:29:37 Redirect to... szilard! 14:29:40 :) 14:30:00 I think it was the deployer creating the patched iso to deploy as root, right? 14:30:09 <[1]JonasB> Okay, szilard status 14:30:11 yes 14:30:36 autodeployer patches the iso file 14:30:43 as usual 14:31:05 <[1]JonasB> szilard: So the behavior hasnt changed 14:31:27 no it is the same as in the prototype 14:32:07 <[1]JonasB> szilard: so it has always had root as owner of the iso file? 14:32:11 Well, the prototype didn't create the ISO as root I think. 14:32:50 <[1]JonasB> Is this an isue or not, Fatih is complaining 14:32:54 Yes that's right 14:33:19 Fatih, what's the problem, still don't understand 14:33:20 yep 14:33:25 this started happening recently 14:33:28 I think it would be good if we don't leave any files owned by root behind after the build, in general. 14:33:54 +1 s_berg 14:34:05 But this is not the build 14:34:15 this is the autodeployer taking the ISO as input 14:34:24 patching it into a temporary iso 14:34:33 then it loads it into the Fuel VM 14:34:37 runs the installation 14:34:43 eject the temporary patched ISO 14:34:47 and removes it completely 14:35:09 and that's it basically 14:35:39 that's how it worked all the time 14:36:04 fdegir: can you show me some error log 14:36:14 <[1]JonasB> So there shouldnt be any thing left with root as owner 14:36:18 szilard: you have it in the mail 14:36:18 yep 14:36:21 and here is the link 14:36:38 https://build.opnfv.org/ci/view/genesis-fuel/job/genesis-fuel-deploy-master/87/console 14:36:42 <[1]JonasB> But Fathi sees residual files with root as owner 14:36:58 szilard: it doesn't remove it 14:37:05 the directory and contents are still there 14:37:17 if you go to pod2 and look at /home/jenkins-ci/workspace/genesis-fuel-deploy-master/fuel/deploy/fueltmp/ 14:37:22 the newiso directory 14:37:34 it could be like you don't remove the stuff if the deployment fails in the middle 14:37:49 Hi, how to down iso image from jenkins? 14:38:03 <[1]JonasB> szilard: Do you remove it if you fail or trap out? 14:38:22 ruijing: isos are stored on http://artifacts.opnfv.org/ 14:38:34 <[1]JonasB> ruijing: You cant, need to download from artifacts 14:38:37 [1]JonasB: yes I remove it if installation fails 14:38:55 fdegir: I dont know right now how it remained there 14:39:03 fdegir: need to investigate 14:39:11 <[1]JonasB> Szilard and Fatih: Can you work this out off-line? 14:39:17 ruijing: you can find latest Fuel ISO for master by downloading http://artifacts.opnfv.org/genesis/fuel/latest.properties 14:39:17 [1]JonasB: sure 14:39:27 [1]JonasB: yes 14:40:44 <[1]JonasB> #info: fdegir has seen ocurance of a patched .iso file owned by root remaining after deploy. Everything owned by root is intended to be removed. szilard and fdegir will work this out 14:40:56 <[1]JonasB> More on deployment 14:40:57 Thanks, I will try it 14:41:16 <[1]JonasB> szilard: Is everything under control? 14:41:34 there are some things to discuss 14:41:44 <[1]JonasB> szilard: Shoot 14:42:14 okay so 14:42:33 The ISO we are building at this moment is 6.1 the plugin is missing 14:42:58 I need to modify the Autdeploy Jenkins job to get the plugin somewhere 14:43:19 or it will not be able to install Opendaylight 14:43:36 all the DEA config files have Opendaylight activated 14:44:01 I create another temporary DEA.yaml config file, where I disable OPendaylight 14:44:06 szilard: Quick Q: It will still deploy though? 14:44:11 szilard: where the plugin is located? 14:44:29 so we can successfully run deployments until we figure out how to get the plugin 14:44:42 fdegir: https://github.com/stackforge/fuel-plugin-opendaylight/tree/juno/lithium-sr1 14:44:44 fdegir: we could download the plugin from Michals 14:44:45 yes 14:44:51 thanks Michal 14:44:55 ok 14:45:04 so then I will update the jenkins job 14:45:12 to get the plugin from that site, okay 14:45:14 maybe jenkins can compile plugin 14:45:17 A question here. Should we not *build* the plugin in the context of OPNFV? 14:45:18 yes szilard 14:45:23 we clone it 14:45:24 build it 14:45:31 <[1]JonasB> szilard: But we should have opendaylight enabled in the CI-deploy 14:46:02 <[1]JonasB> Ahh I see 14:46:05 [1]JonasB: okay I know, that was just a temporary thing, but now we figure out 14:46:07 a way 14:46:12 to get that plugin 14:46:39 so let's discuss how to proceed 14:46:49 <[1]JonasB> Szilard: Cool will you take on the responsibility to get the plugin and install it? 14:47:00 szilard, we can create a separate builder which can clone the plugin src, build it, and put the result to somewhere 14:47:08 yes I can update the jeknins job 14:47:20 under workspace 14:47:20 fdegir: okay 14:47:36 fdegir: ok I will do it, and maybe ask help 14:47:42 if i get stuck 14:47:45 szilard: yep 14:47:49 fdegir: thanks 14:47:58 a question: I suppose we will clone the latest plugin source from github 14:47:59 I also wiped out all the old configuration files 14:48:06 or should we directly start with certain sha1? 14:48:17 https://gerrit.opnfv.org/gerrit/#/c/1645/ 14:48:22 <[1]JonasB> #info Szilard is working on getting the ODL plugin pulled, built and installed 14:48:27 fdegir: I suggest using juno/lithium-sr1 branch 14:48:39 mskalski: okay 14:48:41 mskalski: latest of that branch? 14:48:51 fdegir: yes 14:48:59 ok 14:49:07 if possible, we can use docker to build it 14:49:09 once we verify it works, we tag it and use that instead 14:49:10 <[1]JonasB> So back to the earlier discussion, is Java part of the plugin? 14:49:36 [1]JonasB: No it is not 14:49:46 the thing is 14:49:58 not sure if the build tool has versioning 14:50:13 if we want to distirbuted plugin as an binary 14:50:28 I'm not sure about licensing issues 14:50:37 <[1]JonasB> mskalski: So what is required to get the Java dependency 14:50:54 <[1]JonasB> mskalski: Did you use Oracle Java or open Java? 14:51:51 <[1]JonasB> ruijing: Which build tool are you referring to? 14:52:00 [1]JonasB: openjdk, this is normaly download from ubuntu repository and user does not to do anything, or centos repo added by user 14:52:07 •pip install fuel-plugin-builder 14:52:17 •gem install fpm 14:52:56 not sure if tool including fuel-plugin-builder or fpm has versioning 14:53:03 as decribed here https://github.com/stackforge/fuel-plugin-opendaylight/tree/juno/lithium-sr1#opendaylight-plugin-configuration 14:54:17 <[1]JonasB> mskalski Ahh yes, but does the plugin include the pupet manifests to enable openjdk, ie apt-get install .... 14:55:13 [1]JonasB: yes, actually it is also declared as an dependencies for ODL deb/rpm package 14:55:37 <[1]JonasB> mskalsi: Great then we shouldnt have any issue 14:55:54 <[1]JonasB> Everyone - anything more on SR1? 14:56:10 questino 14:56:13 Hi, JonasB. do you understand me? 14:56:14 if we have repo with java avaiable then no 14:56:27 do we have any extra test cases to be added for SR1 in particular? 14:56:32 or do we just run the standard ones? 14:56:48 <[1]JonasB> Ruijing: Not sure 14:56:52 If build tool chaining is not versioning, we cannot build arno after release 14:57:19 <[1]JonasB> lmcdasm: We run the same ones as for sr0 I believe 14:57:20 build tool like fuel-plugin-builder or fpm 14:58:26 [1]JonasB: There is an issue though... 14:58:27 lets be aware that vm live migration is no supported by ODL, and this health check will fail in case of ceph deployment 14:58:35 <[1]JonasB> ruijing: Yes agree, that isnt optimal. But I guess we will have to store the built SR1 plugins in artifacts. Not optimal 14:58:39 when we build fuel ODL plugin, we don't know what version of fuel-plugin-builder 14:58:43 openjdk needs to be pre-populated, right. 14:59:09 <[1]JonasB> mskalski: Understood 14:59:46 <[1]JonasB> Were reaching the hour. 14:59:55 yes. 14:59:59 <[1]JonasB> Next week we will have to discuss documentation 15:00:05 ok 15:00:24 <[1]JonasB> #info SR1 release date: Sep 28 15:00:48 ruijing: fuel-plugin-builder should be compatible with earlier releases, actually you define in plugin what package version you want to use 15:01:01 <[1]JonasB> So we need to release the channel 15:01:05 <[1]JonasB> 3 15:01:11 2 15:01:20 <[1]JonasB> 1 15:01:20 1 15:01:27 0 15:01:34 *BOOOOM* 15:01:34 <[1]JonasB> #endmeeting