08:10:16 <morgan_orange> #startmeeting Weekly Functest meeting December 6th
08:10:16 <collabot`> Meeting started Tue Dec  6 08:10:16 2016 UTC.  The chair is morgan_orange. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:10:16 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:10:16 <collabot`> The meeting name has been set to 'weekly_functest_meeting_december_6th'
08:10:50 <morgan_orange> #info light meeting as juha and Jose are in UNH for the plugfest and Cedric is at Orange Research exhibition
08:10:55 <morgan_orange> #topic call role
08:10:59 <morgan_orange> #info Morgan Richomme
08:11:05 <HelenYao> #info Helen Yao
08:11:45 <morgan_orange> i do not know if Serena will join
08:11:45 <SerenaFeng> #info SerenaFeng
08:11:53 <morgan_orange> Hello SerenaFeng
08:12:04 <morgan_orange> #topic action points follow-up
08:12:13 <morgan_orange> #link http://ircbot.wl.linuxfoundation.org/meetings/opnfv-functest/2016/opnfv-functest.2016-11-29-08.02.html
08:12:14 <SerenaFeng> #info SerenaFeng
08:12:35 <morgan_orange> #info AP1: HelenYao jose_lausuch continue investigation in Alpine
08:12:46 <HelenYao> no update
08:13:42 <morgan_orange> #info no update
08:13:50 <morgan_orange> #info AP2 morgan_orange create VNF onboarding abstraction class
08:14:01 <morgan_orange> #info work initiate but no patch submitted yet: target this week
08:14:08 <raghav> hi all ..
08:14:13 <raghav> good day
08:14:14 <morgan_orange> #info morgan_orange see with releng if we can customize the jenkins color to distinguish an error due to execution framework from an error due to the tests
08:14:18 <morgan_orange> hi raghav
08:14:37 <morgan_orange> #info question sent to Releng, fatih is processing, no answer yet
08:14:44 <HelenYao> Hi raghav
08:14:49 <morgan_orange> #info AP4 Cedric, propose a patch for TestCaseBase
08:15:06 <raghav> hi
08:15:07 <morgan_orange> #info Cedric is 120% involved in Orange exhibition until the end of the week
08:15:18 <morgan_orange> #info hopefully next week
08:15:29 <morgan_orange> #info AP5 SerenaFeng propose a patch about this topic
08:15:51 <SerenaFeng> no progress
08:16:11 <morgan_orange> #info no progress
08:16:18 <morgan_orange> #info AP6 morgan_orange grant access to rohitsakala to testresults.opnfv.org
08:16:34 <morgan_orange> #info done
08:17:04 <morgan_orange> for this week I planned a light agenda as we are not a lot
08:17:17 <morgan_orange> #link https://wiki.opnfv.org/display/functest/Functest+Meeting
08:17:22 <morgan_orange> do you want to add something?
08:17:31 <morgan_orange> maybe a topic on the refactoring / SNAPS / Helen
08:17:40 <morgan_orange> and a status on the internships?
08:17:50 <HelenYao> sounds good
08:18:42 <morgan_orange> #topic Internal review of landing page proposal (Jack@yardstick)
08:18:54 <morgan_orange> I will start with this one as they are possibily impacts on the test API
08:18:56 <HelenYao> Jack will be here
08:19:01 <morgan_orange> did you get the mail?
08:19:08 <morgan_orange> with the landing page proposal
08:19:14 <morgan_orange> Hi JackChan
08:19:23 <JackChan> Hi Morgan
08:19:42 <HelenYao> sorry, I did not see the mail
08:19:47 <HelenYao> what is the subject?
08:20:26 <morgan_orange> it is about the landing page
08:20:39 <SerenaFeng> opnfv reporting dashboard?
08:20:41 <morgan_orange> the idea is to offer to the community a landing page where it is possible to get the status of the scenarios
08:20:44 <morgan_orange> #link https://postimg.org/image/hqf1vkiun/
08:21:09 <morgan_orange> for the moment Functest has its landing page, Yardstick has its own (both look similar)
08:21:19 <morgan_orange> but when we build a page for a release, we create a wiki page
08:21:26 <morgan_orange> the idea is to create automatically this page
08:21:56 <morgan_orange> so in one glance you can see the status of the scenario
08:22:05 <JackChan> morgan_orange: now I have updated it according to your suggestions.
08:22:13 <HelenYao> the page seems good at first glance
08:22:14 <morgan_orange> it shall be modular to allow new test projects to join the page (e.g. QTIP)
08:22:45 <HelenYao> if more projects join, the page will become very wide
08:22:50 <morgan_orange> JackChan: made several proposals, all were fine but this one was somehow agreed by most of them
08:22:57 <JackChan> Now there is something not done. I will send it to you all later.
08:23:00 <morgan_orange> minor adapations were suggested
08:23:35 <morgan_orange> SerenaFeng: HelenYao raghav with a fresh look: what is your view of this landing page?
08:23:59 <morgan_orange> any comment/remark:suggestion (as said we made some but I do not want to influence...)
08:24:21 <raghav> yeah its good ..
08:24:36 <HelenYao> shall we put all types of installers in the same table? If we want to support more projects, how about making table by installer?
08:24:47 <SerenaFeng> looks very good
08:24:54 <JackChan> Now I have add the filter function. And if more project add in, we can use it to filter.
08:25:14 <morgan_orange> HelenYao: good point, it was one of the remark :)
08:25:15 <HelenYao> yeah, that will be helpful
08:25:34 <SerenaFeng> one question,will the number of test projects increasing, will one line is long enough to have all the test projects?
08:25:37 <raghav> can't we display all type of messages like success/warnings/erros/failures??
08:26:27 <morgan_orange> SerenaFeng: I think the number of test projects will remain reasonable
08:26:44 <morgan_orange> for Danube only QTip and maybe baraometer are news and could lead to additional info
08:26:55 <HelenYao> how about making the 'success', 'danger' into icon instead of using text?
08:27:01 <morgan_orange> the idea was to give the opportunity to all the projects to be representeed,
08:27:19 <HelenYao> make the 'danger' very eye-catching instead of plain text
08:27:22 <morgan_orange> HelenYao: yes we can move from text to icon and maybe find a logo or an icon also for test projects
08:27:38 <SerenaFeng> can the projects be configurable?
08:27:46 <JackChan> SerenaFeng: yes, and i think we can add the function of fold.
08:27:55 <SerenaFeng> for example I only want to have a look at the functest status
08:28:05 <SerenaFeng> I don't want to see all the test projects
08:28:22 <SerenaFeng> JackChan ok
08:28:26 <morgan_orange> I imagine it is possible but we have already the reporting done by each project
08:28:33 <morgan_orange> the goal here is precisely to make a synthesis
08:28:48 <morgan_orange> but on the status of a project you will have hyperlink to current status
08:29:28 <morgan_orange> e.g. yardstick => http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-apex.html
08:29:28 <JackChan> raghav: i don't quite understand you mean.
08:29:45 <HelenYao> what is the page used for? I would say, there is no sign for next step
08:29:48 <SerenaFeng> ok, my misunderstanding, my understanding is this one will substitute the original one
08:30:30 <morgan_orange> no, it will be the landing page, the existing ones will remain
08:30:43 <raghav> they are showing only either warnings or success or dangers
08:30:49 <raghav> instead show all
08:31:16 <morgan_orange> not only the red or green bullet on the scenario indicates if it is validated or not
08:31:37 <morgan_orange> so you can see the list of validated / Trustable scenarios
08:31:44 <morgan_orange> of course there are open questions...
08:32:00 <morgan_orange> do we consider a scenario validated if it is validated on all the installers?
08:32:16 <SerenaFeng> and personally I like plain-text, with icon we need to explain what it stands for
08:32:19 <HelenYao> a trivial question, what is the correct spelling for functest? Functest or FuncTest
08:32:21 <JackChan> raghav: just a example, for sure, we can display all types of message
08:32:43 <raghav> #JackChan : ok
08:33:18 <morgan_orange> HelenYao:  it is not trivial...
08:33:23 <morgan_orange> I used to say functest
08:33:39 <morgan_orange> but I saw peopel writting FuncTest (for Functionnal Testin)
08:34:03 <morgan_orange> another coding name convention problem ... :)
08:34:18 <SerenaFeng> yeah
08:34:23 <SerenaFeng> I use Functest
08:34:34 <morgan_orange> yes functest = repo, Functest = project name
08:34:50 <morgan_orange> #info discussion on the landing page proposal
08:35:00 <morgan_orange> #info landing page does not replace project landing page
08:35:01 <HelenYao> morgan_orange: I thought it's just a demo page, so I did not put the naming convention 1st priority. thx for explanation
08:35:24 <morgan_orange> #info goal at one glance get the status of the scenarios
08:35:46 <morgan_orange> #info suggestion to keep modularity (naming, icons, list of test projects,...)
08:36:02 <morgan_orange> #info still open question : criteria to validate a scenario = f(installer, test project)
08:36:24 <morgan_orange> anyway thanks JackChan, when do you plan to propose a first patch on releng?
08:36:47 <morgan_orange> I think we will need an evolution on the testapi
08:36:57 <morgan_orange> #topic test API and colelction evolution
08:37:13 <morgan_orange> we discussed that in barcelona and in the doc test evolution / CI evolution
08:37:26 <morgan_orange> it is probably time to look at that
08:37:43 <morgan_orange> regarding testcase colelction
08:37:54 <morgan_orange> #info evolution of testcase collection
08:37:57 <morgan_orange> #link https://etherpad.opnfv.org/p/TestCommunityGoals
08:38:02 <JackChan> morgan_orange: Now I have modify it according to your suggestions. And I plan to propose my patch next week.
08:38:05 <morgan_orange> #info several proposals
08:38:45 <morgan_orange> JackChan: cool thanks, we will start working on the test api to introduce the scenario collection (for the moment I assume, Yardstick is processing info locally like Functest)
08:39:07 <morgan_orange> the idea will be also to move information currently in testcases.yaml in the DB...
08:39:34 <morgan_orange> what do you think of the new fields suggested in the pad?
08:39:49 <morgan_orange> I do not undestand "type"..
08:40:05 <morgan_orange> domain and tags are new and could be used to create automatically a test catalog
08:40:23 <morgan_orange> ceretification if for dovetail, so they could flag the case for dovetail or not
08:40:40 <morgan_orange> versions is added to see the compatibility (when the case was introduced)
08:41:16 <morgan_orange> assertion is pushed by bryan to get info on the test procedure, it is not far from the criteria we had in testcases.yaml but we never used
08:41:46 <morgan_orange> for testcase, as it is only new fields, the upgrade of teh DB is not difficult, need to adapt the API + unit tests
08:42:24 <morgan_orange> any remark for testcase?
08:43:17 <HelenYao> do we have the "blocking" field?
08:44:10 <HelenYao> I mean, I did not see it in the DB structure
08:44:15 <SerenaFeng> will we still keep testcases.ymlf?
08:44:23 <morgan_orange> yes we need to add the blocking field
08:44:39 <morgan_orange> we will still keep testcases.yaml in case of DN is not reachable
08:44:44 <morgan_orange> DB
08:44:59 <SerenaFeng> and also there are some information refer to testproject run function
08:45:27 <SerenaFeng> run:
08:45:27 <SerenaFeng> module: 'functest.opnfv_tests.sdn.odl.odl'
08:45:28 <HelenYao> morgan_orange: does it mean, the testcases.yaml will be generated from DB data?
08:45:28 <SerenaFeng> class: 'ODLTests'
08:45:28 <morgan_orange> we could also put this info in the DB
08:46:05 <morgan_orange> HelenYao: ideally yes, I do not know if we will have time to do it, but yes the idea will be to generate the testcases.yaml from the DB, if not possible use a default one
08:46:26 <morgan_orange> the advantage to put the info in the DB is that we can autoamte documentation, domains, visualization,..
08:46:49 <morgan_orange> but it shall be runnable without the DB...(people installing Functest will not always install a Test DB)
08:47:12 <morgan_orange> we can start by adding info and keep the testcases.yaml mechanism
08:47:13 <HelenYao> is there any way to show the tier? in testcases.yaml, it shows the relationship between tier and testcases
08:47:21 <morgan_orange> and implement the generation of the file in E
08:47:37 <morgan_orange> godd point I add tier
08:47:47 <morgan_orange> basically everything in testcases.yaml shall be in the DB
08:47:49 <SerenaFeng> and also we miss ci_loop
08:48:09 <SerenaFeng> to decide it will be executed daily or weekly
08:49:12 <HelenYao> if we want to embrace everything of testcases.yaml into db, the example in the etherpad is not completely showing the details. I would suggest to add a more comprehensive example
08:49:22 <morgan_orange> I would suggest for Danube: put the information (mechanism to override testcases.yaml later)
08:50:18 <HelenYao> so long as we have clear picture about what we want to achieve, we decide which release to include the feature
08:51:21 <HelenYao> one concern about maintaining the copy of testcases.yaml and db at the same time, we have to make them sync at a certain level
08:51:57 <HelenYao> maybe timestamp can be used to refer to, say, lastupdated
08:52:01 <morgan_orange> yes that is why I suggested for D to just override, mechanism to ensure sync + DB offline running needs more work
08:52:27 <HelenYao> yeah, override will be more clean work
08:52:43 <HelenYao> DB > testcases.yaml
08:53:23 <morgan_orange> OK I will action me to clean the etherpad
08:53:38 <morgan_orange> #info discussion on testcases.yaml overwritting
08:54:04 <HelenYao> another concern is, if the testcases.yaml is used by a running testcases, we have to make sure the overwriting does not interrupt the running\
08:54:15 <morgan_orange> #info for Danube focus on more info in DB then see mechanism to use the info from DB (assuming that sync is needed and it shall work offline)
08:54:27 <morgan_orange> #aciton morgan_orange update etherpad and propose a patch
08:54:46 <morgan_orange> #info a new collection "scenario" is also needed
08:55:16 <morgan_orange> for the mioment in testresults.opnfv.org, we recalculate scenario scoring based on results stored in the DB and store the scoring locally in a flat file
08:55:35 <morgan_orange> the goal is to push this info in DB so it could be used by the landing page
08:55:56 <morgan_orange> #topic AoB
08:55:58 <HelenYao> sounds reasonable
08:56:10 <morgan_orange> any other topic you want to discuss?
08:56:14 <morgan_orange> Internships?
08:56:17 <SerenaFeng> will the record of collection 'scenario' is updated or a new one every time?
08:56:29 <morgan_orange> updated
08:56:38 <morgan_orange> with a field results
08:56:59 <morgan_orange> I see a static part and a score fields updated at the end of CI run
08:57:09 <SerenaFeng> and we will keep a certain period of results?
08:57:29 <HelenYao> for the unit test intern project, we had one meeting and Ashish is getting familiar with the Functest framework. He is taking semester exam this week and will work double the time next week
08:57:46 <morgan_orange> good point for the moment we do not have such mechanism, we keep the results since Day 1...
08:58:03 <HelenYao> #info for the unit test intern project, we had one meeting and Ashish is getting familiar with the Functest framework. He is taking semester exam this week and will work double the time next week
08:58:38 <morgan_orange> #info Internship status on VNF catalog to be clarified between Ray and jose during the plugfest (no feedback since agreement but exam period at the moment...)
08:59:12 <HelenYao> #info for the coming sprint, Ashish will work on implementing unit tests for functest/cli, functest/ci
08:59:13 <SerenaFeng> #info Intership status on TestAPI evolution, Rohit is working on beautify the results
08:59:14 <morgan_orange> SerenaFeng: we should probably add a cleaning mechanism on the results if we store the scenario score...
08:59:33 <morgan_orange> HelenYao: regarding Openstack_utils, and your patch
08:59:47 <SerenaFeng> agree, or else the test record will became fatter
08:59:49 <morgan_orange> shall we not consider directly SNAPS
09:00:01 <morgan_orange> as it is already refactored in a object oriented way
09:00:08 <morgan_orange> Steve submited a patch to convert vping
09:00:12 <SerenaFeng> as to the intership, will we calculate their working time?
09:00:30 <HelenYao> I had a check on SNAPS and it is using separate openstack clients, e.g. novaclient, nutronclient
09:00:58 <morgan_orange> HelenYao: can I action you to give a status on that for next week
09:01:01 <SerenaFeng> how we evaluate their work, I think Linux Foundation will ask for a feedback about their work, will they?
09:01:10 <HelenYao> SNAPS has the problem of running on Newton as SNAPS does not support reading the API version for each client
09:01:29 <morgan_orange> SerenaFeng: probably but I think we can keep macro and not track minutes, I will ask ray
09:01:43 <morgan_orange> #action morgan_orange ask ray on the LF expected feedback for the internship
09:02:30 <morgan_orange> #action HelenYao prepare a topic on openstack_utils status for next weekly meeting (refactoring/SNAPS/..)
09:02:46 <HelenYao> ok
09:02:55 <morgan_orange> I should confess I am a bit lost... :(
09:03:04 <morgan_orange> we are already 2 minutes late
09:03:12 <morgan_orange> any other topic before we close the meeting?
09:03:18 <HelenYao> no from me
09:03:22 <SerenaFeng> nope
09:03:26 <morgan_orange> raghav:
09:03:43 <morgan_orange> ok thanks for joining
09:03:48 <morgan_orange> have a good week
09:03:51 <morgan_orange> #endmeeting