08:06:45 <ollivier> #startmeeting Functest weekly meeting 26 June. 2018
08:06:45 <collabot_> Meeting started Tue Jun 26 08:06:45 2018 UTC.  The chair is ollivier. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:06:45 <collabot_> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:06:45 <collabot_> The meeting name has been set to 'functest_weekly_meeting_26_june__2018'
08:06:53 <ollivier> #info Cédric Ollivier
08:06:56 <viktor_t> #info Viktor Tikkanen
08:07:03 <juhak> #info Juha Kosonen
08:08:22 <ollivier> #topic Action point follow-up
08:08:53 <ollivier> #info logs have to be improved (add debug flags)
08:09:57 <ollivier> #info tempest test could run in parallel as for refstack
08:10:20 <ollivier> viktor_t: juhak: what's your feelings about that? I think we have a try
08:10:43 <viktor_t> You mean we have to try?
08:11:27 <ollivier> I would suggest to run them in parallel in master. It seems fine here.
08:11:46 <viktor_t> It seems to be reasonable
08:12:28 <ollivier> thank you.
08:12:39 <ollivier> #info it seems that tempest_full has to be updated (it has to conform with Neutron's tempest full)
08:13:09 <ollivier> I think Neutron doesn't run all tempest tests in gates. I have to check that.
08:14:09 <ollivier> As discussed in LA, I would like to introduce tempest and rally full in all gate. If we can, it would be great to force 100%.
08:14:30 <juhak> +1
08:16:01 <depo> #info Delia Popescu
08:16:08 <ollivier> Regarding a topic discussed last week, a new opnfv/functest-tempest was published. It mainly allows refactoring many Dockerfiles.
08:16:16 <ollivier> Hello Delia
08:16:53 <ollivier> #topic Functest Gambia
08:17:09 <ollivier> #info 6 new scenarios are now offered: tenantnetwork[1,2], vmready[1,2], singlevm[1,2]
08:17:30 <ollivier> it avoids duplicating code as for vnfs
08:17:38 <ollivier> it eases writing testcases for endusers (e.g. shaker: https://gerrit.opnfv.org/gerrit/#/c/58501/)
08:17:43 <ollivier> all testcase config sections will follow the same rules (about overlays, extra flavor specs, etc.)
08:17:47 <ollivier> it could allow easily cleaning all remaining resources
08:18:29 <ollivier> They can be seen as intermediate steps to boot vm reachable via ssh
08:18:51 <juhak> looks great, you have been busy again
08:19:37 <ollivier> Yes but that's not the worst point. You have to know that I'm facing with troubles regarding them.
08:20:22 <ollivier> I added them as intermediate testcases in Healthcheck to allow gating them and to detect the right deployment error
08:21:04 <ollivier> There is a quite huge debate which is overpushed because I added new testcases in healthcheck
08:22:00 <ollivier> It's increased by the fact that I moved odl, cinder and vping into healthcheck as well.
08:22:04 <juhak> I still going through the changes but saw some discussion about that
08:23:35 <viktor_t> are there some scenarios which don't pass healthcheck anymore?
08:24:39 <ollivier> Regarding a global functional scope, I'm even not sure that it changes someting. I do check if one snaps test in api_check asks for metadata.
08:25:54 <ollivier> Badly installers haven't run them since yesterday. I would bet that only sdnvpn could be impacted by that if I understood correctly their tempest issues.
08:26:28 <ollivier> I don't see the point to run smoke if two vms can't ping each others or if ODL api is unreachable. That was my point.
08:27:05 <ollivier> Please let me know your feelings. I can simply revert that changes.
08:27:39 <ollivier> Maybe you need time to read the new classes.
08:27:54 <ollivier> https://gerrit.opnfv.org/gerrit/#/q/status:merged+project:functest+branch:master+topic:scenarios
08:27:58 <ollivier> https://gerrit.opnfv.org/gerrit/#/q/status:merged+project:functest+branch:master+topic:shaker
08:29:27 <ollivier> depo: could you please run healthcheck vs arm?
08:29:53 <depo> It will fail with ping
08:30:18 <juhak> ollivier: will check new classes, but anyway the basic requirement for successful ping test sound reasonable
08:30:36 <ollivier> depo: due to ping or deployment ? Could you please share functest.log
08:30:38 <depo> no, it does not, not for an initial release
08:31:05 <ollivier> juhak: thank you
08:31:11 <depo> we are having noe problems with vm console, mysql issues, is under fix
08:31:19 <depo> console is useless
08:32:08 <viktor_t> It is OK if it is (still) possible to run other tests (like smoke etc.) manually even if healthcheck fails.
08:32:22 <depo> I hear you, but for me healthcheck should mean that openstack is installed and it answers to API checks ( like it has a heart beat.. )
08:32:37 <depo> not ping, at least not in such a short time
08:32:44 <depo> maybe later
08:33:18 <ollivier> healthcheck is designed for the daily build isn't it? We may propose daily releases vs which the enduser can't boot a vm.
08:33:34 <ollivier> If we do so, we have to remove api_check and snaps_healthcheck
08:34:05 <depo> I am not saying that, we will fix the issues by Friday
08:35:09 <depo> you asked my opinion, I say we cannot enforce ping on Monday for Friday
08:35:17 <ollivier> ok. I thought it was about moving vping in smoke again.
08:35:28 <depo> it is
08:36:05 <depo> we cannot enforce anyone to have ping on healthcheck since we have MS on Friday
08:36:36 <depo> it's not about the move itself, it's about the timing
08:36:50 <ollivier> The problem is not milestone here. We can temporarily disable it if needed for Friday.
08:37:20 <ollivier> From an overall point of view, do you agree that vping is part of healthcheck?
08:37:24 <depo> it kind of is the milestone
08:38:12 <depo> later, after MS, we can debate what goes into gating, but we should also implicate installer people and TSC
08:38:37 <depo> and also gating is not about having tests in healthcheck
08:38:58 <depo> it's about what we decie to be reasonable enough criteria
08:39:37 <ollivier> I don't understand how TSC could be involved in testcases selected in Functest tiers. If we follow that Fatih'm model, we are not free to rename or add a testcase.
08:40:52 <ollivier> Even adding one specific test in healthcheck (tenantnetwork which simply configures a tenant network) is raising concerns.
08:41:29 <depo> I agree but installer release /MS depend on functest, we cannot enforce them rules for release either
08:41:59 <viktor_t> There are two sides: a) vping is a very basic functionality and therefore it could belong to healthcheck b) if healthcheck will fail for some reason (e.g. because of ODL problem etc.), the side effect is that other test cases which wwill potentially pass (like most of smoke test cases) will not be executed at all
08:42:36 <depo> we can add/rename tests, but not for the suits that are relase/MS criteria without conulting with people who are impacted by this
08:43:32 <ollivier> a) +1 b) fully true. But it's the same as vpings are the first testcases
08:43:35 <ollivier> in smoke
08:44:46 <ollivier> Maybe atrole may pass even if vping is false. neutron_trunk?
08:44:46 <viktor_t> so if vping will fail in smoke, no other tests will be executed?
08:44:50 <ollivier> yes
08:45:39 <ollivier> We temporarily turn off the mandatory flag when rewriting vping testcases but it has been the config for Functest realeases.
08:45:53 <viktor_t> It was my opinion already years ago that vping is actally quite challenging test case since it requires that everythings works properly :)
08:45:53 <ollivier> I think it's back to mandatory
08:47:07 <depo> yes, it is , but not at MS3.2
08:47:28 <depo> we just ported Queens
08:47:44 <ollivier> My opinion is its the basics requirement for an enduser installing OPNFV. Even if we can tolerate issues for advanced cases (controller)
08:47:47 <depo> at this point is expected not to be very functional
08:48:22 <ollivier> I would propose to disable the test for several installers in failure to be fair.
08:48:48 <ollivier> But moving it to smoke again seem the wrong way.
08:48:52 <depo> it's not fair
08:49:02 <ollivier> Does everybody agree on that?
08:49:22 <ollivier> to be fair regarding milestones.
08:49:39 <depo> why not creayte a separate tier for them?
08:49:44 <depo> *create
08:49:52 <ollivier> A new tier for vping?
08:49:53 <depo> like.. advanced_healthcheck
08:50:08 <depo> advanced_hwalthecheck that requires ping
08:50:35 <viktor_t> Basically I agree with ollivier. But if you will move vping/vping_userdata back to smoke, move them to the end of smoke tests...
08:50:52 <ollivier> We are not able to add a test in healthcheck. I can't imagine what could happen if we propose a new tiers.
08:51:14 <depo> we can try
08:51:59 <ollivier> viktor_t: yes. we will do so.
08:52:17 <ollivier> I will propose the review.
08:52:33 <ollivier> #info connection_check is now leveraging on Shade
08:52:47 <ollivier> #info APEX mster is well verified via tempest_smoke_serial (+Apex promotion snapshot -> functional gating)
08:53:31 <ollivier> About tempest (reused by Dovetail)
08:53:39 <ollivier> we may ease include/exclude tempest subtests via simple regex or service (it would avoid the full lists from Dovetail)
08:53:44 <ollivier> we may add a flag to return False if one subtest is skipped (Dovetail)
08:54:32 <ollivier> #info shaker arm64 image is being shared
08:54:52 <ollivier> depo: I send you a link to send the image. Thank you in advance
08:55:10 <depo> ollivier: sure
08:55:22 <ollivier> #topic AoB:
08:56:18 <ollivier> anything else?
08:56:51 <depo> not from me
08:57:04 <viktor_t> no
08:57:38 <juhak> nothing from my side
08:57:42 <ollivier> Thank you very much for that interesting discussions.
08:57:47 <ollivier> #endmeeting