08:06:18 <morgan_orange> #startmeeting Functest weekly meeting 23rd of May
08:06:18 <collabot`> Meeting started Tue May 23 08:06:18 2017 UTC.  The chair is morgan_orange. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:06:18 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:06:18 <collabot`> The meeting name has been set to 'functest_weekly_meeting_23rd_of_may'
08:06:22 <morgan_orange> #topic call role
08:06:29 <morgan_orange> #info Morgan Richomme
08:06:37 <ollivier> #info Cédric
08:08:01 <morgan_orange> if we are only 2, I think we can cancel....did I miss anything on this meeting?
08:08:19 <SerenaFeng> #info SerenaFeng
08:08:25 <morgan_orange> Hi Serena
08:08:29 <LindaWang> #info Linda Wang
08:08:40 <SerenaFeng> Hi morning
08:09:01 <morgan_orange> Shuya_ool: BTW I am working on vnf refactoring (abstraction class) it may have some impacts. I hope I could submit the patch this week...
08:09:21 <morgan_orange> now we are 3, which is better but a bit light for any discussion
08:09:23 <Shuya_ool> #info Shuya(OOL)
08:09:30 <HelenYao> #info Helen
08:09:58 <morgan_orange> ok seems we hae a quorum now :)
08:10:08 <morgan_orange> #topic action points follow up
08:10:44 <morgan_orange> #link http://ircbot.wl.linuxfoundation.org/meetings/opnfv-functest/2017/opnfv-functest.2017-05-16-08.00.html
08:10:46 <Shuya_ool> morgan_orange: OK, I will do refactor the vyos_vrouter testing.
08:11:01 <morgan_orange> Shuya_ool: so no rush until the patch is available..
08:11:03 <morgan_orange> #info AP1 jose_lausuch work on the 'enabled' flag patcvh
08:11:05 <morgan_orange> #info done
08:11:06 <OPNFV-Gerrit-Bot> Cedric Ollivier proposed functest: Call fetch_os_creds.sh from $PATH  https://gerrit.opnfv.org/gerrit/35169
08:11:16 <morgan_orange> #info AP2 morgan_orange valentin review vnf accoring to framework changes
08:11:18 <jose_lausuch> hi
08:11:20 <jose_lausuch> sorry for the late
08:11:23 <jose_lausuch> #info Jose Lausuch
08:11:26 <Shuya_ool> morgan_orange: Sorry, I got it.
08:11:37 <morgan_orange> #info in progress, hope I can submit something before the end of the week
08:11:47 <jose_lausuch> thanks for starting it
08:12:08 <morgan_orange> #info it will have impact on the existing cases: orchestra_ims, cludify_ims, opera_ims, and future vrouter
08:12:20 <ollivier> LindaWang: Please see the update of https://gerrit.opnfv.org/gerrit/#/c/35169/. Be free to ask any question if unclear.
08:12:27 <morgan_orange> #info but it shall be cleaner and with respect of rules adopted in testcases and features
08:12:41 <morgan_orange> #info AP3 morgan_orange update test case description with new catalog description field
08:12:44 <morgan_orange> #info done
08:12:58 <morgan_orange> #link http://testresults.opnfv.org/test/api/v1/projects/functest/cases
08:13:09 <morgan_orange> feel free to comment if you do not agree on the description
08:13:12 <morgan_orange> yardstick also did it
08:13:26 <morgan_orange> so at least functest and yardstick shall be ready for the case catalog page
08:13:37 <jose_lausuch> cool
08:13:38 <morgan_orange> #info AP4 morgan_orange move from amazon hosting to opnfv resources
08:13:50 <jose_lausuch> there are some test cases from feature projects that still need that field
08:13:53 <morgan_orange> #info done, aricg shall provide the VM this week
08:13:55 <LindaWang> ollivier: Thank you.
08:14:10 <morgan_orange> #info AP5 morgan_orange contact Dave (Urschatz) to see if container suite could be done with functest/cengn mentor
08:14:18 <morgan_orange> #action contact Dave (Urschatz) to see if container suite could be done with functest/cengn mentor
08:14:22 <jose_lausuch> there is an intern
08:14:26 <jose_lausuch> who may work on that soon
08:14:37 <morgan_orange> #info but new intern seems working on that (mentoring narinder)
08:14:56 <morgan_orange> need to xcheck with David, he needs an access to a community lab
08:15:12 <morgan_orange> #info AP6 HelenYao sync with jack Chan for internal API
08:15:18 <morgan_orange> HelenYao: any update on this point?
08:15:33 <HelenYao> Url('/yardstick/asynctask', views.Asynctask, 'asynctask'),
08:15:33 <HelenYao> Url('/yardstick/testcases', views.Testcases, 'testcases'),
08:15:33 <HelenYao> Url('/yardstick/testcases/release/action', views.ReleaseAction, 'release'),
08:15:33 <HelenYao> Url('/yardstick/testcases/samples/action', views.SamplesAction, 'samples'),
08:15:33 <HelenYao> Url('/yardstick/testsuites/action', views.TestsuitesAction, 'testsuites'),
08:15:34 <HelenYao> Url('/yardstick/results', views.Results, 'results'),
08:15:34 <HelenYao> Url('/yardstick/env/action', views.EnvAction
08:15:49 <HelenYao> yardstick has these urls
08:16:01 <HelenYao> we have tier while they have testsuites
08:16:12 <HelenYao> also, they have release/samples for testcases
08:16:48 <HelenYao> do we put tier or testsuites in the api?
08:16:57 <HelenYao> tiers or testsuites
08:16:59 <jose_lausuch> we agreed on tiers...
08:17:05 <jose_lausuch> the ones in the test diagram
08:17:09 <morgan_orange> for me tiers and testsuites are different..tempest, odl are testsuites (i.e. several tests)
08:17:14 <jose_lausuch> ya
08:17:15 <morgan_orange> tiers are just categories
08:17:18 <OPNFV-Gerrit-Bot> Cedric Ollivier proposed functest: Call fetch_os_creds.sh from $PATH  https://gerrit.opnfv.org/gerrit/35169
08:17:18 <jose_lausuch> tiers=categories
08:17:27 <jose_lausuch> test suites=bunch of tests (can be in any category)
08:17:36 <morgan_orange> yep
08:17:41 <morgan_orange> same view
08:17:58 <morgan_orange> so I think their APi coudl be used, minor adaptations needed, no?
08:18:13 <morgan_orange> #chair jose_lausuch
08:18:13 <collabot`> Current chairs: jose_lausuch morgan_orange
08:18:33 <morgan_orange> #info AP7 to publish framework prez'
08:18:39 <HelenYao> i think firstly we have to agree on the functest api definition
08:18:42 <morgan_orange> do not remember what was this poitn?
08:19:02 <morgan_orange> HelenYao: ok, could you make a proposal on the wiki?, so we could validate it next week?
08:19:12 <jose_lausuch> I think it was from ollivier
08:19:31 <SerenaFeng> I suggest to do follow functest cli way, HelenYao make a draft on wiki
08:19:40 <SerenaFeng> then we can review and comments
08:20:02 <HelenYao> ok
08:20:06 <ollivier> Yes. We could define new artifacts for framework prez' and apidoc
08:20:30 <jose_lausuch> but we have to see also what to do with the developer guide
08:20:33 <ollivier> (as now tox generates the api docs)
08:20:39 <jose_lausuch> they are targetting similar purposes
08:20:46 <ollivier> no.
08:21:09 <jose_lausuch> we should avoid duplicating certain information
08:21:10 <morgan_orange> #action HelenYao prepare wiki page for Functest APi proposal (CLI / yardstick approach)
08:22:00 <morgan_orange> for developer guide, I created a developer guide at the testing group level (just a draft for the moment)
08:22:07 <ollivier> I think we have bigger issues than several duplicated data in docstrings, and framework presentation...
08:22:12 <morgan_orange> what is common shall be there, what is specific shall remain on our side
08:22:38 <jose_lausuch> right
08:22:38 <morgan_orange> #info dev guide: work to spli common and functest specific stuff
08:22:50 <jose_lausuch> the test api should be common for example
08:22:52 <morgan_orange> #link https://gerrit.opnfv.org/gerrit/#/c/35179/
08:23:03 <morgan_orange> I will also modify the functest doc
08:23:25 <morgan_orange> but generated doc is really cool, as well as the console view on code quality...
08:23:33 <morgan_orange> good practice to be shared...
08:23:40 <jose_lausuch> agree
08:23:47 <ollivier> It's not a dev guide.
08:23:52 <ollivier> But it makes sense :)
08:23:59 <jose_lausuch> #action SerenaFeng jose_lausuch review https://gerrit.opnfv.org/gerrit/#/c/35179/1
08:24:22 <ollivier> It look like a readme.rst on root of functest
08:24:23 <jose_lausuch> well, we explain the framework, how it works, etc
08:24:23 <morgan_orange> it is really a draft because Jack mentioned it during a mail exchange
08:24:36 <morgan_orange> but that is almost empty, we need to agree on the chapters..
08:24:38 <jose_lausuch> we will care about reorganizing the docs later
08:24:52 <morgan_orange> #info AP7 to create a Jenkins job for api doc
08:24:53 <jose_lausuch> we will have a session with Sofia in the summit
08:24:59 <morgan_orange> #info I think it is done but not enabled
08:25:47 <morgan_orange> for those who did not see the last enhancements in the verify job
08:25:48 <morgan_orange> https://build.opnfv.org/ci/view/functest/job/functest-verify-master/4161/console
08:25:53 <ollivier> Please use docstings as much as possible instead of managing by hand
08:26:00 <ollivier> rst...
08:26:08 <morgan_orange> you can see now that unit tests can be executed offline, tests are executed towards several env (preparation for py3 support)
08:26:26 <jose_lausuch> cool
08:26:45 <jose_lausuch> I like the message at the end:   congratulations :)
08:26:51 <morgan_orange> and all the files have now a mark ... that should help us to converge to a better quality..
08:27:07 <jose_lausuch> cover:  89%
08:27:25 <morgan_orange> ollivier: running it with my changes on vnf leads to the explosion of *_ims (as they are not adapted yet) :)
08:27:51 <morgan_orange> ok back to the agenda then
08:28:05 <ollivier> be careful. there are lots of alse positives in tests and mainly the last 5% is hard :)
08:28:05 <morgan_orange> #topic JIRA management rules
08:28:28 <jose_lausuch> this comes from a conversation I had with Cedric
08:28:40 <jose_lausuch> we should make use of JIRa the same way
08:28:45 <jose_lausuch> but before that
08:28:57 <jose_lausuch> I would like to ask you if it makes sense to create sprints
08:29:06 <jose_lausuch> we didn't use them in the right way in Danube
08:29:19 <jose_lausuch> we probably need to be more attached to milestones
08:29:28 <jose_lausuch> instead of 3-week sprints
08:29:55 <jose_lausuch> what do yiou think?
08:30:23 <morgan_orange> yep more realistic to stick to milestones
08:30:32 <morgan_orange> sprints are a bit artificial
08:30:41 <SerenaFeng> sprints are barely used
08:30:44 <jose_lausuch> they don't make sense for these type of work
08:30:44 <jose_lausuch> ya
08:31:08 <morgan_orange> and less useless work for the PTL
08:31:29 <morgan_orange> #agree JIRA: no sprint / stick on Milestones
08:31:30 <jose_lausuch> we know we have milestones, for example framework ready
08:31:31 <SerenaFeng> agree
08:31:44 <jose_lausuch> so our focus is in finishing the framework
08:31:45 <jose_lausuch> as we are doing
08:31:49 <jose_lausuch> and things like that
08:32:41 <jose_lausuch> when we go towards milestone : test case ready
08:32:50 <jose_lausuch> we need to focus on finishing the test cases
08:32:56 <jose_lausuch> or in parallel
08:33:03 <jose_lausuch> we are  all not working on framework
08:33:14 <jose_lausuch> and another thing
08:33:40 <jose_lausuch> we shouldn't create a bug-jira for something we can fix
08:33:56 <jose_lausuch> I mean, if I find a bug in Functest, it's ok just if I work on the patch
08:34:11 <jose_lausuch> jira-bugs are more for things that need to be remembered
08:34:19 <jose_lausuch> ollivier: would you like to express your opinion on that?
08:35:28 <ollivier> jose_lausuch: sure
08:35:40 <LindaWang> If I find a bug, but I could not solve this. What should I do?
08:35:59 <ollivier> I think there is no need to open JIRA tickets for bugs concerning your testcase in master.
08:36:05 <jose_lausuch> then it's fine to report it I would think
08:36:22 <ollivier> We should tag as bug all fixes we have to backport on stable
08:36:49 <ollivier> As soon as the patch is merged we cherry-pick it to the right stable branch
08:37:27 <morgan_orange> but the question from LindaWang is what to do with bug you cannot solve by yourself..
08:37:36 <morgan_orange> the risk is to forget it
08:37:45 <jose_lausuch> ya, then you can report it
08:37:46 <ollivier> it's fine in this case if you assign to someone else
08:37:57 <LindaWang> Ok.
08:38:00 <morgan_orange> for me it is fine in this case to create a JIRA as well
08:38:11 <jose_lausuch> please remember to put the fix versions field
08:38:22 <morgan_orange> but we do not need to create artificially jira load for things we know we can solve
08:38:26 <ollivier> but be careful about the version (next release here)
08:38:27 <jose_lausuch> exactly
08:38:38 <morgan_orange> JIRA is not a tracker to know our activity (git/gerrit is the right place)
08:38:52 <ollivier> I think we should focus on gerrit and JIRA should be mainly used by external people
08:38:54 <morgan_orange> but if we cannot solve issues, then it is  good way to remember
08:39:07 <morgan_orange> and assign
08:39:07 <jose_lausuch> I will do a cleanup this week I have time
08:39:20 <ollivier> I have done one last days...
08:39:22 <ollivier> too
08:39:23 <morgan_orange> ok so I think we all agree..
08:39:25 <jose_lausuch> I know
08:39:35 <morgan_orange> next topic?
08:39:36 <jose_lausuch> thanks about it
08:39:44 <jose_lausuch> #topic Progress on framework
08:40:04 <morgan_orange> #info as said in action point follow-up: work in progress for VNF part
08:40:04 <jose_lausuch> so, apparently, releng repo was removed from the dockerfile
08:40:06 <jose_lausuch> why?
08:40:32 <ollivier> Yes! It's better to manage it via requirements.txt
08:41:21 <jose_lausuch> mmm
08:41:27 <LindaWang> I am not sure if any other file in releng is required other than fetch_os_creds.sh.
08:41:28 <jose_lausuch> so now it's installed via requirements?
08:41:38 <jose_lausuch> so the only problem is fetch_os_creds...
08:41:40 <jose_lausuch> we can do the following
08:41:48 <jose_lausuch> if only that file is required
08:41:54 <ollivier> Now we can imagine splitting docker containers if we stop cloning them in Dockerfile instead of several requiremetns
08:41:55 <jose_lausuch> we just do it in the jenkins job
08:42:56 <ollivier> The only issues are induced by the fact we will detect all the bugs in setup.py (see snaps, releng...)
08:43:01 <jose_lausuch> and we mandate functest to be created with the RC file as a volume
08:43:11 <jose_lausuch> I saw snaps... thanks for fixing
08:43:17 <jose_lausuch> will you fix releng?
08:43:28 <ollivier> please see https://gerrit.opnfv.org/gerrit/#/c/35183/
08:44:04 <jose_lausuch> ollivier: ah ok, awesome
08:44:08 <jose_lausuch> but what do you think guys?
08:44:18 <ollivier> There is a big issue in releng related to duplicated setup.py but I will fix it when I finish functest
08:44:20 <jose_lausuch> shall we create the container with the RC as a volume? or we leave as it is
08:44:49 <ollivier> It would be better to first manage correctly requirements.txt.
08:45:18 <ollivier> I clean it when I finish tox support but it remains a lot of external projects
08:45:27 <jose_lausuch> ok
08:45:51 <jose_lausuch> more things
08:45:53 <ollivier> then we can imagine switching to Alpine and offer several containers.
08:47:04 <ollivier> But this is the base... of any python project
08:47:04 <jose_lausuch> ollivier: ok, we can think of that later, not sure if it will be ready for this release
08:47:04 <jose_lausuch> LindaWang: you found some issue with tempest -offline
08:47:04 <LindaWang> It seems more reasonable to create the container with the RC as a volume, cause except different testcases run for different installers, there is no need to pass the param INSTALLER_TYPE.
08:47:10 <jose_lausuch> INSTALLER_TYPE is good to have in CI
08:47:21 <jose_lausuch> it's not mandatory for outside OPNFV
08:47:33 <jose_lausuch> but it is interesting to have to collect some information from the installer
08:47:41 <ollivier> You can set INSTALLER_TYPE = debian if you have your own database
08:47:44 <jose_lausuch> but we can prepare the RC file as a volume
08:48:11 <jose_lausuch> if we do so, we don't need fetch_os_creds in the container and we can forget https://gerrit.opnfv.org/gerrit/#/c/35183/
08:48:29 <jose_lausuch> we just need to tweak the jjob
08:48:34 <morgan_orange> that is how it is done with joid today
08:48:39 <jose_lausuch> yep
08:48:42 <LindaWang> OK. The issue is sometimes you do not know the INSTALLER_TYPE.
08:48:56 <ollivier> Of not. This file must be in setup.py
08:48:56 <jose_lausuch> in CI we always know
08:49:15 <LindaWang> I believe functest is not only just for CI.
08:49:17 <ollivier> If a python project project defines scripts it must install them as well
08:49:39 <ollivier> Here by cloning directly git repository, we mask the real mistakes.
08:49:45 <jose_lausuch> ollivier: yes, but if we do it this way, we don't need it in functest then
08:50:04 <jose_lausuch> I mean, fetch_os_creds
08:50:08 <jose_lausuch> I will work on the jjob today
08:50:12 <ollivier> Sure. But still this file exist in repo, it must be installed. .. called or not
08:50:13 <jose_lausuch> to be called only by jenkins
08:50:26 <jose_lausuch> why?
08:50:34 <jose_lausuch> if it's not used by functest, why should we have it?
08:50:34 <jose_lausuch> it
08:50:40 <jose_lausuch> it's not part of the releng modules
08:51:04 <ollivier> It is...
08:51:15 <HelenYao> I think releng is not the right place to fix the problem
08:51:17 <jose_lausuch> it is now, but with the changes in the jjjob, we won't need it
08:51:40 <HelenYao> releng should not be modified just b/c functest needs it in a specific way
08:51:41 <ollivier> ??
08:51:49 <jose_lausuch> HelenYao: if we automate the creation of the cotnainer with the RC file as a volume, we don't need anymore fetch_os_creds in Functest
08:52:05 <ollivier> Please. Releng offer a script. It must installed it.
08:52:08 <jose_lausuch> and we require that in the documentation when crating the container
08:52:23 <jose_lausuch> ollivier: releng offers 1000 scripts, most of them are not used in Functest
08:52:38 <jose_lausuch> for example, to do the upload to artifacts
08:52:44 <HelenYao> i vote for mapping rc file to container
08:52:46 <jose_lausuch> that is done in the JJOB, not by Functest
08:53:15 <ollivier> amazing.. That's a simple packaging task...
08:53:32 <LindaWang> mapping rc file to container +1
08:53:37 <jose_lausuch> let me work on that today and I will add you to the review
08:53:46 <morgan_orange> I am a bit lost, I am not sure we are speaking of the same thing, aren't we?
08:53:51 <ollivier> you don't understand the point.
08:53:59 <ollivier> morgan_orange: yes.
08:54:30 <morgan_orange> as far as I understand the idea is to adopt the joid way to get rc file and get rid of the fetch_os which is used for fuel, apex and compass)
08:54:47 <jose_lausuch> morgan_orange: yes, taht's the point I'm trying to bring
08:55:08 <morgan_orange> the script is not needed, we need only the rc file
08:55:13 <ollivier> but I'm trying to fix a current issue..
08:55:23 <ollivier> and to explain how we deal with requirements.txt
08:55:39 <jose_lausuch> but this current issue will be fixed by that proposal
08:55:41 <ollivier> and setup.py
08:55:49 <jose_lausuch> we won't need fetch_os_creds any more
08:56:09 <HelenYao> i agree with putting the dependencies in requirements.txt
08:56:11 <morgan_orange> there is no other releng requirements?
08:56:17 <morgan_orange> agree also
08:56:20 <jose_lausuch> morgan_orange: no
08:56:28 <jose_lausuch> morgan_orange: only the ones used by the directory modules
08:56:29 <ollivier> And??? As long as the script could called by any opnfv functest, it must be installed when calling setup.py
08:56:51 <ollivier> if not I can simply switch my change to remove os_fetch...
08:57:19 <ollivier> that's simple packaging work
08:57:32 <jose_lausuch> ollivier: but I don't get that, sorry, maybe I'm confused
08:57:38 <jose_lausuch> why do we want to call it?
08:57:39 <morgan_orange> it is good to clean requirement management
08:57:50 <ollivier> Please don't break all the pending work
08:57:53 <HelenYao> i think functest needs to find a way to adopt with releng instead of changing releng to accommodate functest. if mapping the rc file can remove the dependency of releng shell script, we are done here
08:57:55 <morgan_orange> and it is also good to clean the way we are tetrieving the rc files
08:58:29 <morgan_orange> but we should keep all the work done to manage external requirmeents
08:58:35 <morgan_orange> and be cleaner
08:58:38 <jose_lausuch> HelenYao: by mapping you mean having the docker volume or putting it in setup?
08:58:50 <HelenYao> having docker volume
08:58:56 <jose_lausuch> ok
08:59:16 <HelenYao> in that way, fetch_os_rc.sh will not needed by functest
08:59:23 <jose_lausuch> yes, that's my point
08:59:34 <jose_lausuch> refstack also does it this way
08:59:57 <jose_lausuch> fetch_os_creds is a CI thing, only valid for OPNFv, then, we keep it out of functest
09:00:03 <ollivier> I understand. but will you delete the file?
09:00:12 <jose_lausuch> in functest yes
09:00:16 <ollivier> no globally
09:00:21 <jose_lausuch> but it will be still used by CI (releng)
09:00:30 <ollivier> so let it in setup.py please
09:00:30 <jose_lausuch> I will use it in the jjob
09:00:59 <jose_lausuch> ollivier: there are many more scripts in that directory https://git.opnfv.org/releng/tree/utils
09:01:07 <jose_lausuch> why should we just keep 1?
09:01:19 <jose_lausuch> those util scripts are used by jenkins jobs only
09:01:22 <ollivier> because it's designed as external
09:01:24 <HelenYao> i have the same concern as Jose
09:01:36 <jose_lausuch> it's pointless to leave only that script in setup.py
09:01:42 <jose_lausuch> if it's not used by the modules directory
09:02:07 <ollivier> normally releng should install the script and called them from path. As usual, it's cloned and run from working dir.
09:02:16 <ollivier> basic rule.
09:02:41 <jose_lausuch> but now it's intalled by requirements
09:02:48 <ollivier> please read setup.py documentation
09:02:52 <jose_lausuch> which is the right way
09:03:27 <ollivier> I don't want to fix now releng's setup.py I have to finish the framework for the 5th...
09:03:33 <ollivier> Then I will switch to releng
09:03:49 <ollivier> Functest is enough from the time being
09:03:59 <jose_lausuch> ollivier: don't fix that, it's not your job, there are other people as well :)
09:04:20 <morgan_orange> my view is the new mechanism (requirmeent based) is the right one but it is not needed for fetch_os
09:04:44 <morgan_orange> if we need external stuff, we should adopt it (i.e. no more cloning as we were used to do)
09:04:54 <ollivier> It must be installed if jobs stop cloning the repo
09:05:18 <ollivier> jjob could start by a simple pip install
09:06:00 <jose_lausuch> that's another story...
09:06:06 <morgan_orange> it must be installed if jobs stop cloning the repo and if it is used
09:06:30 <jose_lausuch> we need to close..
09:06:37 <jose_lausuch> we can continue offline
09:06:42 <jose_lausuch> #endmeeting