08:00:04 <jose_lausuch> #startmeeting Functest weekly meeting 10 Oct. 2017
08:00:04 <collabot`> Meeting started Tue Oct 10 08:00:04 2017 UTC.  The chair is jose_lausuch. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:00:04 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:00:04 <collabot`> The meeting name has been set to 'functest_weekly_meeting_10_oct__2017'
08:00:10 <jose_lausuch> #info Agenda https://wiki.opnfv.org/display/functest/Functest+5.+Meeting#Functest5.Meeting-10/10(8UTC)
08:00:15 <jose_lausuch> #topic role call
08:00:20 <jose_lausuch> #info Jose Lausuch
08:00:26 <juhak> #info Juha Kosonen
08:00:28 <viktor_t> #info Viktor Tikkanen
08:00:33 <depo> #info Delia Popescu
08:00:41 <jose_lausuch> wow viktor_t! Welcome back!
08:00:56 <viktor_t> tnx
08:01:15 <LindaWang> #info Linda Wang
08:01:42 <xudan> #info xudan
08:02:14 <CristinaPauna> #info Cristina Pauna
08:02:37 <jose_lausuch> morgan_orange: ping
08:03:54 <morgan_orange> #info Morgan Richomme
08:04:44 <jose_lausuch> #topic action points follow up
08:04:48 <jose_lausuch> hi everyone
08:04:54 <jose_lausuch> #info AP morgan_orange grant access to Orange Community Lab 3 (problem is reproducible) to Shuya_ool
08:05:06 <morgan_orange> #info done
08:05:12 <jose_lausuch> #info AP jose_lausuch fix cleaning docker mechanism
08:05:14 <morgan_orange> #info Shuya_ool can reproduce the issue
08:05:39 <jose_lausuch> #info I did a cleanup in the build servers that are used for Docker
08:06:09 <jose_lausuch> #info but a better way to clean leftovers should be in place by infra
08:06:20 <jose_lausuch> #info AP  morgan_orange check nighlty run / vping_userdata and check why Daisy run is PASS (https://build.opnfv.org/ci/view/functest/job/functest-daisy-baremetal-daily-euphrates/) run supposed to integrated the reporting fix but not the router fix
08:06:46 <morgan_orange> #info analyzis done by Cedric , in soem case it coudl work (woudl explain why it was OK on Daisy)
08:07:03 <jose_lausuch> ok, thanks
08:07:13 <jose_lausuch> I remember we had to add a router always
08:07:20 <jose_lausuch> since Arno...
08:07:26 <jose_lausuch> #info AP  morgan_orange contact Daisy team to get details on metadata management
08:07:27 <morgan_orange> #info adding a router fix the problem in any case
08:07:52 <morgan_orange> oops did not do that (holidays in China anyway)
08:08:05 <morgan_orange> #action morgan_orange contact Daisy team to get details on metadata management
08:08:11 <jose_lausuch> ok
08:08:18 <jose_lausuch> #info AP  juhak check if referenced Danube upstream bugs (motivating tempest black listing) are still valid or not
08:08:25 <juhak> #info apex and compass: these are ok, no blacklisting needed
08:08:35 <juhak> #info fuel: one related bug report still open. Will clarify if blacklisting needed, no answer from Michael yet
08:08:43 <juhak> #info joid: not related to bug but I think some blacklisting related to lxd scenarios is needed as in Danube. Asked confirmation from Narinder, no answer yet
08:09:03 <jose_lausuch> juhak: but you said that blacklist is not applied in CI, right?
08:09:33 <juhak> blacklisting is there but list is empty
08:10:08 <LindaWang> juhak: I am afraid some testscases have to be blacklisted  by Compass soon if they can not be fixed.
08:10:11 <jose_lausuch> juhak: ok, I know the file is blank. but the mechanism still works I imagine
08:10:32 <jose_lausuch> LindaWang: not sure that is a good approach :)
08:10:40 <juhak> jose_lausuch:  that's right
08:11:04 <morgan_orange> blacklisting can be enabled only if upstream bug is clearly referenced
08:11:12 <jose_lausuch> morgan_orange: exactly
08:11:29 <jose_lausuch> but not just because something fails and can't be fixed
08:11:32 <juhak> LindaWang: ok, just let me know when that's needed
08:11:33 <LindaWang> Ok, got it.
08:11:53 <jose_lausuch> CI testing should reflect what's working and what's not
08:11:56 <jose_lausuch> ok
08:12:04 <jose_lausuch> thanks juhak
08:12:06 <jose_lausuch> #info AP  jose_lausuch initiate the release note
08:12:19 <jose_lausuch> #info will do it next week since there is still some time for the release
08:12:27 <jose_lausuch> #info AP  morgan_orange initiate an etherpad to collect F features
08:12:33 <morgan_orange> #info done
08:12:41 <jose_lausuch> can you paste the link?
08:12:42 <morgan_orange> #link https://wiki.opnfv.org/display/functest/F+page
08:12:45 <jose_lausuch> thx
08:12:52 <morgan_orange> #info already first page and gating initiated by Cedric
08:13:09 <morgan_orange> I just added an etherpad for discussions => see next topic
08:13:13 <jose_lausuch> morgan_orange: I created Fraser page
08:13:17 <jose_lausuch> under releases
08:13:23 <jose_lausuch> https://wiki.opnfv.org/display/functest/6.+Fraser
08:13:32 <jose_lausuch> maybe we can remove F page in root
08:13:35 <morgan_orange> OK I can put everything under Fraser, as we know the name now
08:13:42 <morgan_orange> I remove the F page
08:13:46 <jose_lausuch> ok
08:13:50 <jose_lausuch> thanks
08:14:12 <jose_lausuch> #info AP  morgan_orange ollivier prepare a presentation on Alpine / requirements management (slot booked on the testing group first) then possible to share with TSC
08:14:42 <morgan_orange> #info in progress..I started, I will commit today or tomorrow a first version
08:15:00 <jose_lausuch> ok
08:15:01 <jose_lausuch> thanks
08:15:14 <jose_lausuch> #topic Release status
08:15:35 <jose_lausuch> #info Release is on the 20th October
08:16:10 <jose_lausuch> #info this way, scenarios get more iterations and there is more time to see stability
08:16:21 <jose_lausuch> (or unstability)
08:16:31 <morgan_orange> for me we are ready...(except the release note :)) and should already ask for branching...to indicate to release manager..that we were on time..
08:16:45 <jose_lausuch> branching?
08:16:48 <morgan_orange> oops
08:16:50 <morgan_orange> tagging
08:16:55 <jose_lausuch> well
08:16:58 <jose_lausuch> I don't agree 100%
08:17:14 <jose_lausuch> if we tag now and build our release images, what if other projects do bug fixes?
08:17:18 <jose_lausuch> we won't include them
08:17:23 <ollivier> Yes. I fully agree
08:17:23 <jose_lausuch> there is still time for bugfixing
08:17:31 <jose_lausuch> and we don't need to rush
08:17:41 <jose_lausuch> tagging can wait when everyone does it
08:17:45 <jose_lausuch> in my opinion
08:18:09 <ollivier> My opnion is we are mixing opnfv & functest gating. Functest is ready and the tag can be asked.
08:18:39 <jose_lausuch> is there a milestone for tagging?
08:19:00 <morgan_orange> not sure, to be clarified
08:19:11 <ollivier> Technically speaking we should wait the last opnfv projects but there will be still bugs in projects. That's why I prefer using stable/euphrates and then the tag can be safely asked.
08:19:30 <ollivier> Functest was ready on time.
08:19:33 <jose_lausuch> yes there is
08:19:36 <jose_lausuch> we can't do whatever we want
08:19:47 <ollivier> We can't blame Functest too
08:19:56 <jose_lausuch> this is not a race, we don't need to show that we were on time by tagging
08:20:02 <jose_lausuch> nobody is blaming functest
08:20:32 <ollivier> The mail is clear. We can ask for the tag when ready.
08:20:37 <jose_lausuch> what mail?
08:20:50 <morgan_orange> blaming is maybe a bit strong, but David's mail sounds a little bit like that. And it is mainly due to the fact that there is a confusion between Functest and OPNFV release criteria
08:21:31 <morgan_orange> there is no rush, I agree, it is just a way to clearly distinguish both for the future...
08:21:41 <jose_lausuch> but I think your answers were also clear
08:22:02 <jose_lausuch> I think we should wait for the final tag
08:22:18 <jose_lausuch> otherwise, we won't include the last status of the feature tests
08:22:29 <ollivier> https://lists.opnfv.org/pipermail/opnfv-tech-discuss/2017-September/017851.html
08:22:49 <jose_lausuch> agree we could use euphrates tag, and it's documented, but the release tag will still be opnfv-5.0.0 or whatever
08:23:21 <morgan_orange> I think you coudl say in the release meeting that Functest is ready for tagging...and we will follow the rules. But it is important to report that een if all the scenarios are not OK, it is not linked to Functest.. :)
08:23:22 <jose_lausuch> ollivier: this mail talks about branching
08:23:23 <ollivier> In fact this is a dream. The tag won't be perfect and there will be bugs in features project. Fucntest is ready for opnfv-5.0.0
08:23:36 <ollivier> YEs I'm mixing :)
08:23:44 <jose_lausuch> morgan_orange: I Can say that, of course, but we have to follow the release rules
08:23:51 <morgan_orange> ok
08:24:08 <jose_lausuch> and I also said that 2 weeks ago btw
08:24:09 <jose_lausuch> :)
08:24:18 <jose_lausuch> and repeated another time to David
08:24:37 <jose_lausuch> again, this is not a race or a contest to see who is on time and who is not
08:24:45 <morgan_orange> repetition is the baseline for pedagogy ...as long as david is using Functest dashboard for release criteria you may say it again ...
08:25:14 <morgan_orange> of course there is no race
08:25:22 <jose_lausuch> and the reason for delaying Euphrates was mainly "to get more iterations"
08:25:24 <morgan_orange> but one of the issue is the milestones that nobody really follows
08:25:39 <jose_lausuch> so, that's the CI process, no one blamed us :)
08:25:53 <jose_lausuch> that's another discussion… yes
08:25:56 <ollivier> jose_lausuch: regarding your last point It's not related to any race. Simply about Functest quality. Look at the gerrit reviews.
08:26:56 <jose_lausuch> yes, but I guess there are other projects that also ready and they are not asking for tagging :)
08:27:40 <CristinaPauna> guys, if you really want to tag anything, I think you can use release candidates
08:27:59 <CristinaPauna> can we discuss a bit how to go forward with alpine arch?
08:28:11 <CristinaPauna> It is still unclear to us
08:28:21 <jose_lausuch> we can tag, sure, but not use the release tag now :)
08:28:21 <CristinaPauna> do we go with manifest or with tag?
08:28:42 <jose_lausuch> #topic Alpine arch
08:28:57 <jose_lausuch> good topic to be discussed, thanks CristinaPauna
08:29:06 <ollivier> We could start working on F. I don't understand why we can't tag.
08:29:55 <depo> we can set the architecture in the tag
08:30:11 <depo> i understnd that now the images are build through dockerhub
08:30:59 <depo> so if we have a manifest for functest-core, like cedric pruposed, we would only need a patch for funtest-core, right?
08:31:02 <jose_lausuch> ollivier: it doesn't mean we can't start on F, master is for continuous improvement, the tag will be anyways done in stable/euphrates btranch
08:31:03 <depo> for arm
08:31:26 <depo> and all dockerfiles for testcases and restapi would have the same image source functest-core:multiarch
08:31:31 <depo> for arm and x86
08:31:37 <ollivier> jose_lausuch: Partially false. It means we can mostly focus on F instead of E as we have done the job.
08:31:40 <depo> and any other arch...
08:32:58 <ollivier> Using manifest will avoid all patches.
08:33:21 <depo> all but one, for functest-core...
08:33:25 <depo> or am I wrong?
08:34:18 <ollivier> Yes.
08:34:47 <ollivier> I think Alpin is not multiarch (x86 is official)
08:35:13 <ollivier> I will write a full wiki page on the topic.
08:35:34 <depo> no, it's not multiarch.
08:35:43 <depo> we would have to have a multiarch image manifest
08:35:47 <depo> for functest-core
08:36:00 <ollivier> Yes as it was in ollivier repo
08:36:08 <depo> yes
08:36:39 <depo> and my sec questions is , how will the manifest be build ?
08:36:45 <depo> automatically?
08:36:48 <ollivier> There is now rush now as we are still working for E and it's too late for this release.
08:37:20 <depo> last week you said that we should do alpine for E release...
08:37:37 <ollivier> Yes. It must be done automatically but releng is not ready. I could add hocks for that but I would prefer to wait the jenkins jobs.
08:37:39 <depo> is it no longer the case now?
08:38:18 <ollivier> I said we should have done.
08:38:37 <depo> so we will do the build for the testcases with Dockerhub and for functest-core and img manifest with jenkins job?
08:38:52 <ollivier> jenkins jobs are not ready.
08:39:11 <ollivier> That's the main issue.
08:39:23 <jose_lausuch> ollivier: what do we need from jenkins jobs exactly? enable the alpine builds?
08:39:55 <ollivier> Manage dependencies
08:40:22 <depo> and I guess multijob
08:40:43 <jose_lausuch> ok
08:40:45 <depo> and make sure that functest-core is build before the images for the testcases
08:40:55 <ollivier> IT's already welll explained in reviews.
08:41:45 <depo> can you please explain the plan for this process? how we should continue?
08:42:03 <ollivier> But we have already checked the technical part from a functest point of view (cross build, manifest...)
08:42:45 <ollivier> Thank you for the access.
08:42:48 <jose_lausuch> ollivier: can we use the script you did and improve it to trigger the docker builds from jenkins using that script? https://git.opnfv.org/functest/tree/build.sh
08:43:42 <ollivier> I would say no. I don't understand why the topic has been abondonned in the middle of the release.
08:45:02 <ollivier> Why opnfv clones my tmp ollivier docker automated build.
08:45:35 <jose_lausuch> opnfv clones my tmp ollivier docker automated build?
08:46:43 <ollivier> Yes. I simply fixed the first ones, complete with the last containers. We should have worked on jenkins ways to build them. The last proposals came too late  and were wrong.
08:47:22 <depo> do we have anyone working on jenkins build?
08:47:45 <depo> or a plan on how to build them? since we have build dependences
08:47:48 <jose_lausuch> I am not the main one but I can help if needed
08:48:11 <ollivier> I thinik releng should manage that are they are more skilled on that.
08:48:35 <jose_lausuch> infra team
08:48:54 <jose_lausuch> let's continue after the meeting, there are other topics
08:49:04 <jose_lausuch> #info Dovetail Euphrates
08:49:07 <jose_lausuch> #undo
08:49:07 <collabot`> Removing item from minutes: <MeetBot.ircmeeting.items.Info object at 0x3760890>
08:49:09 <jose_lausuch> #topic Dovetail Euphrates
08:49:17 <jose_lausuch> xudan: ping
08:49:22 <xudan> Yes
08:49:31 <jose_lausuch> #info creation of their own functest-cvp in Dovetail? or
08:49:38 <jose_lausuch> #info just host config (yaml) and trigger the adhoc dockers (smoke + features+components)
08:50:11 <jose_lausuch> xudan: can you explain your concerns about this?
08:50:29 <xudan> I think create functest-cvp in Dovetail is better.
08:50:46 <jose_lausuch> yes, and you can maintain in
08:50:51 <jose_lausuch> I think it makes sense
08:51:04 <jose_lausuch> you can have your own testcases.yaml, with only the ones you need for Dovetail testing
08:51:17 <morgan_orange> as the dockers already exist, the creation of an additional one would not bring added values, would it?...creating the 3 yaml files (smoke+features+components) is easier - no docker maintenance
08:51:17 <jose_lausuch> this is not changing the code or anything, just stating the test cases you want to execute
08:51:31 <jose_lausuch> that's the other option, using what we already have
08:51:48 <jose_lausuch> xudan: consider also morgan_orange comment
08:51:48 <xudan> If Dovetail wants to use E release functest, it's better to use functest Alphine images rather than the ubuntu image
08:51:56 <jose_lausuch> xudan: yes
08:52:06 <morgan_orange> xudan: Ubuntu image will no more be supported in F
08:52:23 <morgan_orange> so yes considering alpine dockers is the right way
08:53:05 <morgan_orange> possible to tag them with specific cvp tag, then in your repo you "just" put the yaml file
08:53:10 <xudan> The dockerfile for functest-cvp and the testcases.yaml can be hosted by Dovetail.
08:53:16 <morgan_orange> and when running them you replace the defualt yaml by your yaml
08:53:31 <jose_lausuch> xudan: you have to evaluate how complex is to call functest from Dovetail. If using 1 container is easier for you, go for it, but if you can just use what we already have, it's maybe less work for you
08:54:29 <xudan> But one question. How can we build this docker image?
08:54:44 <jose_lausuch> xudan: we are saying that you might not need a new docker image :)
08:54:47 <xudan> I think using 1 container is easier for Dovetail.
08:55:16 <ollivier> xudan: that differs from our target.
08:55:25 <LindaWang> jose_lausuch: why don't they need a new docker image?
08:55:54 <jose_lausuch> LindaWang: because everything what Dovetail needs is already part of our docker images, the only thing is customization (testcases.yaml)
08:55:55 <LindaWang> jose_lausuch: They could use 3 images (smoke, features and components)?
08:56:06 <ollivier> LindaWang: agree
08:56:26 <jose_lausuch> xudan: what test cases are you using from functest?
08:56:29 <morgan_orange> LindaWang: agree (that is what I triying to say..: ))
08:56:47 <xudan> refstack, vping, sdnvpn and tempest_custom
08:57:09 <xudan> Maybe there will be more test cases...
08:57:10 <jose_lausuch> so for components, you can have your own testcases.yaml that calls tempest_custom and don't include rally-full and temepst-full
08:57:12 <jose_lausuch> that's just an example
08:57:28 <LindaWang> But in that case, is it convenient to use 3 images respectively according to different testcases?
08:57:38 <ollivier> yes
08:57:40 <jose_lausuch> yes
08:57:51 <morgan_orange> sudo docker run --env-file env -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds   -v dovetail-smoke.yaml:/usr/lib/python2.7/site-available/functest/ci/testcases.yaml  -v $(pwd)/images:/home/opnfv/functest/images  \    opnfv/functest-smoke:cpv.5.0
08:58:02 <morgan_orange> same for features and components
08:58:04 <jose_lausuch> for example
08:58:20 <jose_lausuch> you only need to override the default testcases.yaml with the one you will host in Dovetail
08:58:45 <jose_lausuch> I think it's simpler and you don't need to maintain a new docker image
08:58:49 <morgan_orange> I added the -v in the example =>  -v dovetail-smoke.yaml:/usr/lib/python2.7/site-available/functest/ci/testcases.yaml
08:59:04 <morgan_orange> and the only think to do is create a yaml file
08:59:11 <morgan_orange> 3 one for each tier
08:59:14 <jose_lausuch> well, 3 yamls, yes
08:59:18 <jose_lausuch> exactly
08:59:27 <jose_lausuch> xudan: how does it sound?
08:59:35 <LindaWang> xudan: But you mentioned in the email, changing testcases.yaml will also compromise the impartiality of Dovetail?
08:59:36 <ollivier> or 3 sed commands to disable the other testcases if you prefer.
08:59:37 <jose_lausuch> we can support you in that task
09:00:14 <morgan_orange> ollivier: they need to add tempest_custom
09:00:26 <xudan> Yes, change testcases.yaml is not very official for Dovetail
09:00:36 <LindaWang> I can not understand why do we have to remove tempest_custom in testcases.yaml. It does not influence much
09:00:42 <ollivier> morgan_orange: yes sed ca rebuild the world
09:00:51 <jose_lausuch> xudan: testcases.yaml is just a way to select the testcases you want to run, that's it
09:01:28 <xudan> I know.
09:01:28 <LindaWang> And it is very convenient to debug tempest_smoke_serial issue, as users can add the specific testcases into tempest_custom.
09:01:43 <ollivier> morgan_orange: it can insert a global section or can rename a previous testcase definition ;)
09:01:54 <xudan> But from the aspect of Dovetail, the user may ask why you change Functest source code.
09:02:05 <ollivier> testcases.yaml is the default file for gating Functest.
09:02:17 <ollivier> It's normal that it's written for that.
09:02:27 <xudan> To the user, they don't take it as a config file. they think it's functest source code.
09:02:40 <LindaWang> I just realized that tempest_custom is also important for installer team to debug, as least Compass team prefer to run it.
09:02:41 <ollivier> etmpest_custom is unused. But the python code is still there. you're free to run it.
09:02:59 <jose_lausuch> we are out of time
09:03:17 <jose_lausuch> xudan: can you bring that topic to Dovetail and come back with feedback?
09:03:19 <LindaWang> tempest is just not used by CI
09:03:28 <xudan> Yes. it's important for debuging
09:03:37 <ollivier> All file hosted by Functest should be written for Functest...
09:03:38 <LindaWang> It doesnot mean that it is not used by other users
09:03:38 <jose_lausuch> the purpose of testcases.yaml is to be customized for users, it's not changing the way Functest behaves
09:03:49 <morgan_orange> then you can create your testcases-debugging.yaml and overwrite the existing one
09:03:51 <jose_lausuch> it's just a declaration of the test cases you wanna run
09:04:11 <ollivier> you're free to rewritten logging.ini too for instance.
09:04:52 <jose_lausuch> #action xudan bring topic of custom testcases.yaml to Dovetail team
09:04:53 <xudan> I will discuss with my team and let you know.
09:05:03 <juhak> LindaWang: just received vpn, thank you!
09:05:04 <jose_lausuch> xudan: ok, thanks
09:05:05 <xudan> Thanks everyone.
09:05:25 <jose_lausuch> #endmeeting