07:59:40 <ollivier> #startmeeting Functest weekly meeting 21 Aug. 2018
07:59:40 <collabot> Meeting started Tue Aug 21 07:59:40 2018 UTC.  The chair is ollivier. Information about MeetBot at http://wiki.debian.org/MeetBot.
07:59:40 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
07:59:40 <collabot> The meeting name has been set to 'functest_weekly_meeting_21_aug__2018'
07:59:41 <viktor_t> Hi!
07:59:52 <ollivier> #info Cédric Ollivier
07:59:59 <viktor_t> #info Viktor Tikkanen
08:00:03 <juhak> #info Juha Kosonen
08:00:17 <depo> #info Delia Popescu
08:01:38 <ollivier> #topic Status
08:01:51 <ollivier> #info Functest Fraser successfully verified by Apex
08:01:59 <ollivier> #link https://build.opnfv.org/ci/view/functest/job/functest-apex-baremetal-daily-fraser/265/console
08:02:13 <ollivier> #infp Functest Gambia successfully verified by Apex and Fuel (amd64)
08:02:25 <ollivier> #info Functest Gambia successfully verified by Apex and Fuel (amd64)
08:02:35 <ollivier> #link https://build.opnfv.org/ci/view/functest/job/functest-apex-baremetal-daily-master/210/console
08:02:46 <ollivier> #link https://build.opnfv.org/ci/view/functest/job/functest-fuel-baremetal-daily-master/397/console
08:03:00 <ollivier> #info Functest Hunter successfully verified by Apex
08:03:07 <ollivier> rally_sanity failed due to a missing feature: live-migration
08:03:13 <ollivier> snaps testcases are disabled as they can't verify Cinder Rocky
08:03:21 <ollivier> #link https://build.opnfv.org/ci/view/functest/job/functest-apex-baremetal-daily-master/205/console
08:03:36 <ollivier> #info Functest Kubernetes Fraser: no run
08:03:46 <ollivier> #info Functest Kubernetes Gambia successfully verified by Compass
08:04:03 <ollivier> #link https://build.opnfv.org/ci/view/functest/job/functest-compass-baremetal-daily-master/257/console
08:04:21 <ollivier> Great Job! Everything seems working!
08:04:22 <georgk> #info Georg Kunz
08:05:09 <ollivier> Now all Functest K8s containers are built vs arm64 as well.
08:06:38 <ollivier> depo: it seems Fuel arm64 is not successfully verified right now (Gambia). I think it's out of Functest, no?
08:07:13 <depo> yes
08:07:58 <ollivier> The key question: do we switch to OpenStack Release schedule?
08:08:44 <ollivier> it requires to implement Functional gates (+ parallel runs) to ease maintaining all that branches but the results seem ok.
08:09:29 <ollivier> It's very difficult to maintain requirement synchronization asked by XCI and APEX master without the right branches.
08:10:12 <ollivier> (see ollivier repositories)
08:10:55 <ollivier> no comment?
08:11:02 <viktor_t> is this switching to the release schedule somehow risky?
08:12:35 <ollivier> If we don't have the right CI/CD, I would think yes because of the extra work (but it's already the case)
08:12:49 <ollivier> Technically speaking Functest is already working in all that cases.
08:14:21 <juhak> are there any identified potential drawbacks for functest users?
08:14:41 <ollivier> The model won't work as soon as we set COMMIT_ID in Gambia (or we will override lots of DOCKER build args)
08:15:08 <ollivier> I would think no but it could induce lots of cherry-picks for us.
08:15:40 <viktor_t> For me the switching is OK if it eases maintenance significantly...
08:16:03 <juhak> agree that
08:16:07 <ollivier> a new feature developed in Functest master will be cherry-picked in Functest Gambia.
08:16:40 <ollivier> ok. Let's continue working on functional gates then that scenario is possible. Thank you.
08:16:59 <depo> agree
08:18:04 <ollivier> Another key point (already discussed a little bit) is to conform with OpenStack Gates.
08:18:04 <viktor_t> cherry-picking is probably needed only for most critical features/bug fixes?...
08:18:21 <ollivier> do we include tempest_scenario (30 min), tempest_full (1h) or rally_full in gates (3h) in gates? At least no issue if parallel
08:20:00 <ollivier> I haven't checked yet which rally testcases are run in OS Gates .
08:20:31 <ollivier> #link http://logs.openstack.org/78/591978/6/check/neutron-rally-task/3bb1d82/
08:20:41 <juhak> maybe rally_full (3h) is too long
08:21:53 <viktor_t> tempest_* cases can be included if they are stable (can be run successfully few times in a row for the same deployment)
08:22:32 <ollivier> There are stable from a OS point of view. But we haven't run them vs OPNFV installers.
08:23:23 <ollivier> Maybe we may add tempest_scenario and tempest_full in all gates (they are equal to OS's tempest full).
08:24:01 <ollivier> APEX and Fuel are already stable. We may have quick results.
08:24:37 <ollivier> jjb timeout is 8h and Apex ends in 4h if I'm not wrong.
08:25:46 <depo> yes, this is known to be an issue
08:26:18 <ollivier> juhak: viktor_t: could you please check if rally_smoke is equal to the OpenStack CI jobs. http://logs.openstack.org/78/591978/6/check/neutron-rally-task/3bb1d82/
08:26:45 <juhak> ok, will do
08:27:24 <ollivier> When I ran everything in parallel, my jjb end in more or less the same duration (4h). rally full included.
08:27:27 <ollivier> thank you.
08:29:32 <ollivier> I noticed that we should update Snaps testcases to do so (to create a new user as for Rally) https://jira.opnfv.org/projects/FUNCTEST/issues/FUNCTEST-1003
08:30:13 <ollivier> be free to assign to yourself if you're interested in.
08:30:38 <juhak> I can take that
08:30:52 <ollivier> thank you.
08:31:41 <ollivier> Then it has to be backported in Fraser to allow gating fraser and master in // as well.
08:32:33 <OPNFV-Gerrit-Bot> Stamatis Katsaounis proposed functest: [WIP] Run Intel NFV CI test-cases inside functest smoke.  https://gerrit.opnfv.org/gerrit/61151
08:33:57 <ollivier> FYI, I tried running functest fraser, master and ollivier containers in //. The results were not so bad (few failures mostly due kolla-ansible deployment).
08:35:14 <ollivier> I crashed Jenkins several times due to the number of executors :)
08:36:24 <ollivier> last questions: do we create a dedicated container for benchmark tools? smoke seems more or less the OpenStack gates (tempest + rally)
08:36:38 <ollivier> could we remove useless subdirs (opnfv_tests, openstack, etc...)?
08:36:52 <ollivier> could we remove all tiers? it's now useless and would allow refactoring Xtesting
08:38:29 <depo> can we like do a tets in another environment first?
08:39:03 <juhak> a dedicated container for benchmark tools sounds good to me
08:39:22 <depo> for me too
08:40:24 <ollivier> depo: I don't see your point about another environment . About parallel testing?
08:40:37 <depo> yes
08:41:15 <depo> I can try to do that on fuel arm
08:41:31 <ollivier> That's what I'm doing. I'm testing parallel testing in Orange Onap Openlab. But all that changes are required for sequential and parallel testing.
08:42:00 <ollivier> If we create specific users, we wont' face with cidr conflicts, quota issues due to remaining resources
08:42:43 <ollivier> Thank you. Could you please try the new jjbs. You only have to edit globals.yaml. You can load all of them easily
08:42:57 <depo> yes, I agree with you, but we may have different behavior on different env
08:43:34 <depo> I can try
08:43:44 <ollivier> sure. And the key issues could be possible admin operations in tempest or rally which could break another testcase.
08:44:34 <ollivier> But it could highlight possible misconfigurations about workers and it will save lots of time.
08:45:05 <depo> but jenkins executors won't be an issue?
08:45:54 <ollivier> in releng yes. Here it tested vs 10 without troubles. I tried up to 30 but it requires much more memory
08:47:05 <depo> so, we'll just live them in functest?
08:47:37 <ollivier> The new jjbs can be easily integrated by endusers. There are not linked to OPNFV.
08:48:43 <ollivier> Why not at the end, offering a docker compose config tdeploying jenkins, the jjbs and the database.
08:49:08 <depo> ok
08:49:55 <ollivier> But if parallel testing works well, we will ask to update releng again.
08:50:33 <ollivier> I was hoping in weekly testing too but I haven't seen any progress.
08:51:36 <ollivier> #topic key actions before release
08:51:38 <depo> ok. I will try parallel testing on arm too
08:51:47 <ollivier> thank you
08:51:52 <ollivier> #update all docs (+publish them online)
08:52:02 <ollivier> #info update all docs (+publish them online)
08:52:15 <ollivier> #info fix live tests in heat_ims
08:52:43 <ollivier> We could add helper to convert qcow2 in vmdk + VIO like patch configs
08:54:39 <ollivier> We could develop basic tests to cover Swift and Heat.
08:54:47 <ollivier> #link https://jira.opnfv.org/browse/FUNCTEST-962
08:54:55 <ollivier> + one pending dev on crypto as well
08:55:30 <juhak> are all installers supporting to install and configure Swift into use?
08:56:22 <ollivier> I don't think so. But we can verify Switf endpoints before running the testcase.
08:56:43 <juhak> we could also add Swift test cases to rally scenarios, if needed
08:57:41 <ollivier> Yes it would be great. but we may skip it on-the-fly as OPNFV doesn't set any rule regarding the OpenStack services.
08:58:01 <juhak> ok
08:58:35 <ollivier> https://xtesting.readthedocs.io/en/latest/apidoc/xtesting.core.testcase.html#xtesting.core.testcase.TestCase.check_requirements
08:58:50 <ollivier> thank you.
08:58:53 <ollivier> #topic AoB
08:59:20 <ollivier> #action ollivier advertise the new framework to OPNFV Features et all
08:59:33 <ollivier> #action ollivier check the release plan of OPNFV Features integrated in Functest
09:00:13 <ollivier> All Functest features are synced to OpenStack Queens but sdnvpn is running on Apex master.
09:00:26 <ollivier> It's fully unclear.
09:00:32 <ollivier> Anything else?
09:00:49 <depo> not from me
09:01:12 <viktor_t> no
09:01:15 <juhak> ollivier: thank you for keeping me posted about vmtp corrections you have upstreamed
09:03:56 <ollivier> juhak: I think the upsteam work is over. Maybe we could help about arm64 binaries.
09:04:54 <ollivier> We are a little bit late. Have a great week. Thank you.
09:05:08 <depo> have a great week
09:05:12 <ollivier> Functest is working very well (even Hunter :))
09:05:55 <ollivier> #endmeeting