08:00:10 <ollivier> #startmeeting Functest weekly meeting 21 Nov. 2017
08:00:10 <collabot> Meeting started Tue Nov 21 08:00:10 2017 UTC.  The chair is ollivier. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:00:10 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:00:10 <collabot> The meeting name has been set to 'functest_weekly_meeting_21_nov__2017'
08:00:31 <ollivier> #topic role call
08:00:46 <jose_lausuch> #info Jose Lausuch
08:00:53 <viktor_t> #info Viktor Tikkanen
08:00:56 <LindaWang> #info Linda Wang
08:01:14 <ollivier> #info Cédric
08:01:57 <ollivier> #topic action points follow up
08:02:10 <ollivier> #info vEPC still under development (the container seems good)
08:02:51 <ollivier> I think it's key for 5.1. Maybe we should help them.
08:03:40 <LindaWang> i forgot to answer the email about vpec issue on tested locally on Compass
08:03:48 <ollivier> else I'm not sure we will integrate it on time. Can anyone help on that point?
08:04:20 <jose_lausuch> I can't help much in that point
08:04:36 <LindaWang> ollivier: sorry, me  either
08:04:52 <ollivier> I think you already sent an useful email. Do you have something else to share?
08:05:19 <ollivier> ok. Let's see if they will update it.
08:05:31 <LindaWang> actually not so far, i did not investigate it deeper
08:06:38 <ollivier> I haven't seen any update of  "ims testing for Heat Orchestration in mano" https://gerrit.opnfv.org/gerrit/40889
08:06:54 <ollivier> I will send an email before abandonning it.
08:07:11 <ollivier> #info one new bug related to OS_AUTH_URL
08:07:58 <LindaWang> #link https://gerrit.opnfv.org/gerrit/#/c/47553/
08:08:03 <ollivier> Could you please review https://gerrit.opnfv.org/gerrit/#/c/47553/ ? Then we should cherry-pick it to E.
08:09:07 <jose_lausuch> looks ok
08:09:23 <ollivier> About Releng:
08:09:25 <OPNFV-Gerrit-Bot> Merged functest: Fix the way of getting endpoint port  https://gerrit.opnfv.org/gerrit/47553
08:09:27 <ollivier> #info multijobs were fixed (we should implement the container garbage collector)
08:10:21 <jose_lausuch> or improve it
08:10:26 <jose_lausuch> I can work on that
08:10:44 <ollivier> on multijobs?
08:10:51 <jose_lausuch> on garbage collector
08:10:55 <jose_lausuch> cleaning containers
08:11:01 <jose_lausuch> I've been doing that manually on the build servers
08:11:18 <LindaWang> no container garbage collector so far?
08:11:23 <ollivier> great. Could you also add a variable for the repo? I would be useful for preprod
08:11:41 <jose_lausuch> what do you mean?
08:11:48 <jose_lausuch> LindaWang: there is, but no so efficient
08:11:54 <ollivier> #action jose_lausuch to complete Functest multijobs
08:12:12 <LindaWang> jose_lausuch:  which shell script do you refer to?
08:12:17 <jose_lausuch> shall we still do the clean at the beginning?
08:12:27 <ollivier> opnfv is hardcoded. why not reusing the multijobs for ollivier repositories as well?
08:13:01 <LindaWang> jose_lausuch: or in the end, either is fine
08:13:09 <jose_lausuch> ok, I will contact you these days to do that
08:13:12 <ollivier> #agree
08:13:20 <jose_lausuch> there is a problem if we do it in the end, if the job fails it will quit
08:14:23 <LindaWang> jose_lausuch:  oh, do we  need to create a new shell script?
08:14:49 <ollivier> Yes. But we do take care of the list before cleaning (if functest-core is listed or not)
08:14:49 <LindaWang> compass team is improving the images cleanup: https://gerrit.opnfv.org/gerrit/#/c/47329/7/jjb/releng/opnfv-docker.sh
08:14:57 <LindaWang> just FYI
08:15:05 <ollivier> I think we could implement it in build.sh ?
08:15:12 <jose_lausuch> I didn't see that
08:15:22 <jose_lausuch> ollivier: yes, that would make sense
08:15:25 <LindaWang> ollivier:  AGREE
08:15:45 <ollivier> Let's go for that. And we will discuss on the review.
08:15:51 <jose_lausuch> ok
08:15:59 <ollivier> #info we got rid of the former Dockerfile and disabled the Docker automated builds
08:16:30 <jose_lausuch> cudos for that
08:16:53 <ollivier> Great news. I will implement a wiki page to list pros/cons of the different solutions used in Functest (releng/travis-ci/docker automated build...)
08:17:18 <jose_lausuch> ok
08:17:39 <ollivier> About Fraser:
08:17:48 <ollivier> #info all features are being synchronized with OpenStack Pike
08:18:10 <ollivier> #info sdnvpn and functest-odl-sfc testcases were enabled for F-release
08:18:18 <ollivier> #info refstack were updated to 2017.09 and we selected the subset of snaps testcases
08:18:45 <ollivier> #info rally-manage db create is deprecated (work in progress)
08:19:03 <LindaWang> #link https://gerrit.opnfv.org/gerrit/#/c/47555/ under review
08:19:21 <ollivier> Yes. Could you please review this patch too?
08:19:34 <jose_lausuch> sure
08:19:59 <jose_lausuch> printf?
08:20:12 <ollivier> yes. That's the best tool here :)
08:20:32 <ollivier> #info we're approaching MS3.0 in a few weeks (Dec 15)
08:21:00 <ollivier> #info Healthcheck suite seems successful for compass, fuel and apex (no run for 4 days)
08:21:43 <ollivier> Badly Serena isn't here. I think it doesn't work in case of Daisy.
08:22:15 <jose_lausuch> I think Apex is not running master
08:22:18 <jose_lausuch> or is it?
08:22:35 <ollivier> At least 4 days ago.
08:22:47 <jose_lausuch> no
08:22:47 <jose_lausuch> https://build.opnfv.org/ci/view/apex/job/apex-daily-master/
08:22:52 <jose_lausuch> last run September 26th
08:22:53 <ollivier> I reported to David that Functest is fine regarding the timeslot
08:23:38 <ollivier> Does everyone agree?
08:23:49 <LindaWang> #link https://build.opnfv.org/ci/job/apex-build-master/ Apex has issue with deployment
08:24:20 <jose_lausuch> ollivier: yes
08:24:27 <ollivier> LindaWang: I noted that snaps_smoke failed in compass.
08:24:45 <LindaWang> i will help check that
08:25:29 <ollivier> #action LindaWang to check why snaps_smoke failed in compass
08:25:50 <ollivier> About Functional Gating/Xtesting
08:26:05 <ollivier> #info Dockerfile is being published for SFC: https://gerrit.opnfv.org/gerrit/#/c/47505/
08:27:01 <ollivier> I think Functest is close to be ready for Functional Gating but we have to wait for XCI and SFC (F?)
08:27:12 <jose_lausuch> but SFC is not ready to test with XCI
08:27:23 <jose_lausuch> they said a few weeks/months
08:27:29 <jose_lausuch> if lucky, before Xmas
08:27:53 <ollivier> yes. And Fatih said that we have to wait Zuul v3 for the workflow vote.
08:28:54 <ollivier> But I think it's a good target even if we are too early. We could continue on sharing Dockerfiles on top of functest-core for snaps and Features.
08:29:05 <jose_lausuch> yes
08:29:18 <LindaWang> we will not provide Snaps dedicated container for patch gating, right?
08:29:19 <jose_lausuch> that's nice
08:29:24 <ollivier> It's quite trivial for us. The only issue is to maintain all of them.
08:30:03 <jose_lausuch> ollivier: maybe for trivial changes like openstack release update, it can be done automatically
08:30:04 <ollivier> Agree. In case of snaps we could get the status of our container even if it's non voting.
08:30:32 <LindaWang> ok
08:31:06 <ollivier> jose_lausuch: yes. I am not afraid about that point. It's just an issue when multiplying the Dockerfiles.
08:31:34 <jose_lausuch> yes, we will see
08:31:47 <jose_lausuch> too early to propose automation maybe, but good to think about that
08:31:54 <jose_lausuch> SFC is the first one doing the container thing
08:31:57 <ollivier> I would be great if we received email if one snaps change breaks healthcheck even if we don't block the change automatically.
08:31:59 <jose_lausuch> maybe we could try with SDNVPN as well
08:32:44 <ollivier> yes. We had to wait for the update of requirements.
08:33:35 <ollivier> #action ollivier propose Dockerfile and testcases.yaml for sdnvpn too.
08:33:53 <ollivier> #info rally/tempest had been moved to the right containers
08:34:32 <ollivier> Thank you for that. The size of functest-core should have decreased.
08:35:21 <jose_lausuch> according to dockerhub only 67 MB ...
08:35:30 <ollivier> divided by 2 :)
08:35:31 <jose_lausuch> vs Euphrates wich was 122 MB
08:35:39 <ollivier> (It's compressed)
08:35:46 <jose_lausuch> that's right
08:36:15 <jose_lausuch> 220MB uncompressed
08:36:27 <jose_lausuch> vs 340 euphrates
08:36:35 <jose_lausuch> very light
08:36:37 <LindaWang> yes,
08:36:41 <ollivier> #info one issue introduced by then update of rally https://gerrit.opnfv.org/gerrit/#/dashboard/self
08:36:47 <jose_lausuch> our ubuntu image took around 1 GB or more
08:37:02 <ollivier> yes :) But several images were included too.
08:37:40 <ollivier> Now any feature can build a container in less than 1 minute.
08:37:52 <jose_lausuch> great achievement
08:38:31 <ollivier> Yes and I think we can't reduce that. xtesting could decrease by removing OpenStack dependencies. But functest-core will still contain them.
08:38:52 <ollivier> and all OPNFV features will still depend on functest-core
08:39:18 <ollivier> bravo Functest!
08:39:26 <LindaWang> will openstack dependencies be included in functest-core?
08:39:33 <jose_lausuch> :)
08:39:36 <LindaWang> i am still confused by functest-core and xtesting
08:39:42 <LindaWang> the difference
08:39:51 <jose_lausuch> LindaWang: right time to ask :)
08:40:17 <ollivier> Yes. I will intiate an email thread about that. I haven't written enough data about that. Sorry.
08:41:03 <ollivier> Sorry I should have done that last week. Functest-core contains everything for running OPNFV testcases.
08:41:14 <ollivier> (hosted by Functest)
08:41:59 <LindaWang> what about functest/utils ?
08:42:04 <ollivier> Xtesting will contain only the Framework + entry points. All abstract classes (TestCase, Feature, Vnf) without any link to OpenStack.
08:42:48 <jose_lausuch> can we keep functes tutils in core?
08:43:15 <ollivier> Yes. great points. If utils are used by run_tests, they should be in xtesting. If they are useful to operate OpenStack resources, they must in Functest (then Functest-core)
08:43:58 <jose_lausuch> functest_utils are not openstack related, but openstack_utils are
08:44:08 <LindaWang> Are ""functest snapshot create and cleanup  still useful now?
08:44:14 <jose_lausuch> but openstack_utils doesn't have good quaility
08:44:33 <jose_lausuch> LindaWang: we don't use that in CI any more
08:44:44 <LindaWang> shall we remove that?
08:45:03 <jose_lausuch> we could, although it's a nice tool for the user
08:45:38 <LindaWang> but as i know, dovetail is using it for clean the resources left by sdnvpn
08:45:49 <ollivier> I think we could switch to snaps or openstack utils but that cannot be achieved in one change.
08:46:01 <LindaWang> but since sdnvpn has improved its resources cleanup, we could remove it
08:46:05 <ollivier> We have to check if tempest cleans all its resources now.
08:46:17 <LindaWang> ollivier: it seems thta
08:46:28 <jose_lausuch> sdnvpn has a JIRA to create their own openstack utils, since we are deprecating it
08:46:58 <LindaWang> what about sfc?
08:47:08 <LindaWang> are they using the os_utils, too?
08:47:20 <jose_lausuch> I think so
08:47:47 <ollivier> I think it's good target to remove openstack utils. Let's try to achieve that and see if they are blocking issues.
08:47:55 <LindaWang> ollivier: currently tempest has not used openstack_cleanup
08:48:50 <ollivier> LindaWang: yes. we worked on that for E.
08:50:00 <ollivier> I think Functest should decide what to do with our code. We should remove libraires at the beginning of the release in case it breaks features.
08:50:43 <ollivier> Maybe we could try removing snapshots/clean as we are at the beginning of the release, no?
08:51:16 <jose_lausuch> ollivier: yes, I think it won't have side effects as we are not using it for CI
08:51:38 <LindaWang> even for the service clients, will we also want to get them by SNAPS?
08:52:05 <jose_lausuch> look at rally how it is done
08:52:08 <jose_lausuch> it's more clean
08:52:13 <ollivier> #action everyone to check if we can remove snapshots/clean in all TestCases.
08:52:59 <ollivier> #action everyone to remove openstack utils calls in our TestCases
08:53:02 <ollivier> agree?
08:53:09 <LindaWang> but then, we will have to spend more energy to take care of snaps too.
08:53:43 <ollivier> maybe we could use OpenStack clients for basic tasks.
08:53:45 <LindaWang> i am afraid maybe snaps' bug will block us from time to time
08:54:12 <jose_lausuch> if we don't do so, we will have to spend a lot of energy in our opensatck utils
08:54:13 <LindaWang> which basic tasks?
08:54:19 <jose_lausuch> and all the code using that...
08:55:09 <ollivier> It's fine to set this target. Let's see if possible. I won't be done in one change as rewriting python packages and Dockerfiles.
08:55:32 <ollivier> #info we are closed to remove prepare_env: first approach is to move the remaining code (ARM related) to run_tests
08:56:16 <jose_lausuch> yes
08:56:19 <jose_lausuch> I am working on the UT
08:56:21 <ollivier> I think we have to work on UT. impl and jjobs are close to be ready.
08:56:21 <jose_lausuch> it's not easy...
08:56:34 <ollivier> Yes. I know.
08:56:34 <jose_lausuch> my patch is ready to be tested, LindaWang can you help ?
08:56:50 <LindaWang> i have tested it
08:56:58 <LindaWang> and i have given some comments
08:56:59 <jose_lausuch> oh thanks
08:57:05 <ollivier> #action jose_lausuch LindaWang to complete the two patches about prepare_env
08:57:16 <ollivier> #info new topic: to switch to ODL NetVirt suites instead of the basic suites no longer used by ODL
08:57:24 <jose_lausuch> ok, talk to you later about it
08:57:45 <ollivier> I will forward an email I received from ODL.
08:58:02 <jose_lausuch> ollivier: great, we should also sync with SDNVPN guys, they have ideas about more tests for ODL
08:58:15 <jose_lausuch> which are not vpn related
08:58:18 <ollivier> jose_lausuch: yes.
08:58:29 <jose_lausuch> it's more ODL only, and we don't know who should own those tests
08:58:33 <jose_lausuch> since there is no ODL project
08:58:38 <jose_lausuch> we can discuss in emails
08:58:56 <ollivier> yes. It can be hosted by Functest.
08:59:07 <jose_lausuch> +1
08:59:20 <ollivier> #topic AoB
08:59:49 <jose_lausuch> nothing else from my side
08:59:56 <LindaWang> If we want to support testcases run on non-CI deployments,  how to make testcases independent on INSTALLER_TYPE and DEPLOY_SCENARIO?
09:00:14 <LindaWang> ollivier:  you mentioned , we could support testcases.yaml editable?
09:00:19 <LindaWang> what does that mean?
09:00:23 <jose_lausuch> isn't that possible already?
09:00:28 <LindaWang> do you guys have idea?
09:00:44 <jose_lausuch> you can have your own testcases.yaml and replace it as a docker volume
09:00:45 <ollivier> For my personal testcase, I set Debian as INSTALLER_TYPE and it works well.
09:00:46 <jose_lausuch> if that helps
09:01:12 <LindaWang> so just edit testcases.yaml , is enough?
09:01:13 <ollivier> From a global point of view, we shouldn't link Functest to CI
09:01:17 <jose_lausuch> but I think dependencies shouldn't be removed, as they indicate the possibility to run only on certain installers/scenarios
09:01:23 <LindaWang> ollivier: i agree
09:01:38 <LindaWang> the files i functest/ci seem work only for CI
09:01:46 <LindaWang> the files iN functest/ci seem work only for CI
09:02:18 <ollivier> My feeling is we must provide a basic testcases.yaml for all third-parties. It will be great to modify the file by one http call (one testcase to define).
09:02:32 <LindaWang> For no-opnfv deployments, no such INSTALLER_TYPE and DEPLOY_SCENARIO at all.
09:03:07 <jose_lausuch> what happens if you run functest without INSTALLER_TYPE and DEPLOY_SCENARIO ?
09:03:13 <ollivier> SCENARIO could be relevant anyway.
09:03:54 <LindaWang> jose_lausuch:  we have not checked thta, because DEPLOY_SCENARIO is onsdn-nofeature-noha by default
09:03:56 <jose_lausuch> DEPLOY_SCENARIO is a way to filter tests … if no set, it is difficult to see which test cases can be run
09:04:12 <ollivier> Any change breaking links to CI data is welcome!
09:04:29 <jose_lausuch> maybe it should be mandatory, as in the future we will support k8 deployments, and that is not os-nosdn-nofeature
09:05:40 <ollivier> Great meeting! I propose to close it we are a little bit late but simply because of the great work done this week ;)
09:05:49 <ollivier> #endmeeting