14:25:17 <rprakash> #startmeeting Dovetail 14:25:17 <collabot> Meeting started Fri Jul 29 14:25:17 2016 UTC. The chair is rprakash. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:25:17 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic. 14:25:17 <collabot> The meeting name has been set to 'dovetail' 14:25:23 <dneary> rprakash, I just proposed postponing... 14:25:39 <matthewli_> #info Jun Li 14:25:48 <rprakash> OK if everyone agrees I will call it off 14:25:53 <yuyang> #info yuyang 14:26:08 <dneary> #info Dave Neary (intermitten network access) 14:26:10 <matthewli_> For me it's Ok 14:26:14 <dneary> #undo 14:26:18 <dneary> #info Dave Neary (intermittent network access) 14:26:38 <dneary> Although intermitten is a nice word... 14:26:41 <ChrisPriceAB> #info Chris 14:26:48 <rprakash> #info rprakash 14:27:20 <rprakash> # topic Continuation of Discussions on use cases SFC, L3VPN 14:27:46 <rprakash> #ifo Dave had some questions on this and lets start with his inputs 14:27:52 <rprakash> undo 14:28:06 <rprakash> #info Dave had some questions on this and lets start with his inputs 14:28:20 <rprakash> Goahead Dave 14:28:20 <ChrisPriceAB> Can we start with a statement of what we have? (been away) 14:28:23 <dneary> rprakash, your #topic won't register 14:28:23 <dneary> You just undid your #info rprakash 14:28:48 <dneary> rprakash, Can you chair me and Chris please? 14:29:02 <rprakash> how do i do that? 14:29:03 <dneary> ChrisPriceAB, I will recap for the minutes 14:29:14 <dneary> Just #chair ChrisPriceAB dneary 14:29:20 <ChrisPriceAB> thanks dneary 14:29:39 <rprakash> #chair ChrisPriceAB dneary 14:29:39 <collabot> Current chairs: ChrisPriceAB dneary rprakash 14:29:51 <dneary> #topic Continuing discussions on L3VPN, SFC use-cases 14:30:07 <rprakash> goahead chris recap 14:30:25 <dneary> #info Chris Donley attended the Dovetail meeting on behalf of the C&C Committee 2 weeks ago 14:30:58 <dneary> #info His input to the Dovetail discussion was that the C&C Committee wanted L3VPN, IPv6 overlay, and SFC to be part of the scope for Dovetail testing 14:32:04 <dneary> #info There were significant concerns expressed about the feasibility of including any of those features in scope for conformance testing in a Colorado timeframe (dneary, Bryan Sullivan, David McBride) 14:32:07 <ChrisPriceAB> These might be better questions for the C&C call. but did he indicate in what context these use cases should be tested? 14:32:27 <dneary> No 14:32:32 <dneary> Not to my knowledge 14:32:56 <rprakash> #info There was discussions on API vs. Use case Approach 14:33:05 <ChrisPriceAB> ok, thanks. Tend to agree with the concerns expressed. 14:33:21 <ChrisPriceAB> rprakash, what is meant by API Vs Use Case? 14:34:03 <dneary> #info The main concerns were that Dovetail tests should pass at least with multiple installers, controllers, and scenarios inside OPNFV before we consider including them in a test suite which should apply to other VIMs, SDN controllers, etc 14:34:10 <rprakash> #info using ETSI NFV TST APIs vs Blackbox approach where we use cases as stated 14:34:46 <dneary> #info and SFC does not have a method-independent test yet (there are VNFFG tests and SFC tests), IPv6 overlay does not work end to end yet, and L3VPN is still in development 14:35:15 <rprakash> #dave yes passing test is important but we do not have APIs in very project exposed like in OpenStack Modules so saying we need to teat ETSI API based complaince is too far 14:35:27 <dneary> #info It was minuted from the last meeting that there was agreement on including these features in scope, and dneary noted in an email that it was not agreed. 14:36:11 <dneary> rprakash, #info, #question, #topic at the start of a sentence are the main meetbot commands - anything outside that (for the most patr) is ignored (FYI) 14:36:16 <matthewli_> #info IPv6 overlay is almost ready, while with some hardcoded not fixed, however, it's the C release time, code freezed, 14:36:25 <dneary> ChrisPriceAB, You're caught up 14:36:54 <ChrisPriceAB> Hmm, OK. I think there is a fundamental question of what is required to achieve a dovetail test case. 14:37:01 <matthewli_> After talked with Hubin and working in Yardstick 14:37:04 <rprakash> #info I am not saying use cases is agreed but at a broader level C&C said that in essence of time to satrt testing on release C we must start with use cases instead of APIs like tempest like suite' 14:37:35 <ChrisPriceAB> how do we test SFC if we have not yet specified a test for the ready state of the platform. 14:37:49 <ChrisPriceAB> It seems like an unlikely test to provide any value at all to the industry, 14:38:03 <ChrisPriceAB> same could be said about other use cases. 14:38:08 <rprakash> #info Chris agreed that's what Dave had concerns and we were to discuss that 14:38:27 <ChrisPriceAB> #link http://lists.opnfv.org/pipermail/opnfv-tech-discuss/2016-July/011725.html e-mail form chris proposing establishing test case criteria 14:38:41 <rprakash> #info • Since API from OPNFV projects are yet to be compiled and gap assessed with respect to ETSI TS APIs for many projects that have MANO implications, the baseline currently appears to be OPENSTACK + SDNC and NB APIs are going to take time to settle. 14:38:59 <ChrisPriceAB> In my mind we need to establish, a) a set of needed tests, b) qualifications for tests, c) outputs expected 14:39:19 <ChrisPriceAB> needed tests do not start with VPN. They start with "are the lights on?". 14:39:54 <dneary> ChrisPriceAB, I agree 14:40:22 <ChrisPriceAB> We can test a lot of capability from a VNF perspective. This seems like the most important set of suites that would enable the ecosystem we are trying to incubate of application developers. MANO does not help them and is premature. 14:40:26 <rprakash> #info the Functest and Yardstic tests do that and C&C suggested adding or stressing on IPv6 (Overlay) , SFC and L3VPN 14:40:35 <dneary> #info ChrisPriceAB states that the starting point should be basic test about the state of the platform - "are the lights on?" - and L3VPN is out of scope for that 14:40:51 <ChrisPriceAB> Well, is not the starting point at least. ;) 14:41:05 <dneary> rprakash, Functest and Yardstick are not compliance test suites though 14:41:26 <dneary> rprakash, We need to extract some of the functest tests as the core of the Dovetail test suite, IMHO 14:41:28 <ChrisPriceAB> #info I also think that we can leverage test cases from other test project ONLY AFTER we have defined the requirements of a dovetail test case. 14:42:00 <dneary> I proposed requirements for a Dovetail test suite 14:42:12 <dneary> * Must pass on multiple installers 14:42:20 <rprakash> #info OK let us agee that the start point is Functest and Yardstick and lets select out of it what can we use as harness for Dovetail 14:42:21 <dneary> * With multiple SDN controller back-ends 14:42:30 <dneary> * Running multiple scenarios 14:42:32 <ChrisPriceAB> no, no the test suite. requirements on a "test case". 14:42:41 <dneary> s/suite/case 14:42:55 <dneary> ChrisPriceAB, Apologies - my mistake. I meant test case 14:43:20 <ChrisPriceAB> I had proposed a list like this in my e-mail: 14:43:23 <ChrisPriceAB> · Test cases should implement a published standard interface for validation 14:43:23 <ChrisPriceAB> o Where no standard is available provide API support references 14:43:23 <ChrisPriceAB> o If a standard exists is not followed an exemption method needs to be established 14:43:23 <ChrisPriceAB> · Test cases be documented 14:43:23 <ChrisPriceAB> o Use case specification 14:43:24 <ChrisPriceAB> o Test preconditions 14:43:24 <ChrisPriceAB> o Basic test flow execution descriptor 14:43:25 <ChrisPriceAB> o Post conditions and pass fail criteria 14:43:25 <ChrisPriceAB> o Parameter border test cases descriptions 14:43:26 <ChrisPriceAB> o Fault/Error test case descriptions (this feels optional at this time) 14:43:26 <ChrisPriceAB> · Test documentation/implementation file and directory structure (per supported framework) 14:43:26 <dneary> I think "multiple" can be as little as 2 for the installers and SDN controllers 14:43:27 <ChrisPriceAB> · Test result storage, structure and test result information management (these should be able to be run publically or privately) 14:43:49 <ChrisPriceAB> we need also to define return types expected for success and failure etc. 14:43:56 <ChrisPriceAB> logginf requirements. 14:44:43 <ChrisPriceAB> This is for a compliance/certification activity, not a development testing activity. The requirements will need to be quite high on tests, behaviours, and test documentation. 14:45:01 <dneary> ChrisPriceAB, I agree on that last point 14:45:14 <dneary> Still working through your requirements list... 14:45:18 <matthewli_> Agree 14:45:23 <rprakash> #info Chris I agree this for Compliance Verifciation Program and not Dvelopment testing 14:46:00 <rprakash> #info ltes talk published standards 14:46:00 <dneary> rprakash, Also, if we could keep Meetbot for minutes and avoid using it to record conversation, it will make for better minutes 14:46:54 <ChrisPriceAB> #info Maybe we can spin up a wiki page under dovetail to walk through our test case requiremenents. Happy to start that. 14:47:03 <rprakash> #info #opnfv-dovtail needs meetbot update once that is done recording will be easier 14:47:12 <dneary> #info ChrisPriceAB and dneary proposed criteria for consideration of test cases as part of Dovetail 14:47:22 <rprakash> #info until then we use this channel 14:47:44 <ChrisPriceAB> No, rprakash 14:48:09 <ChrisPriceAB> The community has indicated that we do not want to spin up per project IRC chanels. If you have a desire to do so please adress the TSC prior to doing so. 14:48:14 <dneary> rprakash, I'm not talking about the channel, I'm talking about how you get good minutes using MeetBot. 14:48:49 <dneary> Just as you don't minute everything that is said on a conf call, MeetBot should be for summarising the meeting, not recording everything that is said. There are full logs for that. 14:49:00 <rprakash> #info lets continue using this channel, there were conflicts on irc use last time and hence I suggested new irc channel with meetobot 14:49:07 <dneary> #undo 14:49:07 <collabot> Removing item from minutes: <MeetBot.ircmeeting.items.Info object at 0x2847c90> 14:49:32 <dneary> rprakash, That did not need to be minuted, for example 14:49:58 <rprakash> Ok i will avoid using info for sundry items 14:50:08 <ChrisPriceAB> if we have problems here the testperf chanel would seemt o be the most logical next chanel to use. 14:50:23 <ChrisPriceAB> where were we? 14:51:00 <rprakash> you mean #opnfv-testperf as long as there is no conflic on time usage its fine with all participants 14:51:34 <dneary> ChrisPriceAB, Your criteria for test cases confuses me a little - the standards section: how will that work with something like OpenStack APIs? 14:51:45 <ChrisPriceAB> this is a test project, we should use meeting, if not lets defer to testperf 14:51:54 <rprakash> we were at Test cases should implement a published standard interface for validation 14:52:14 <dneary> There are standards that exist and are ignored in the space, do we need to get an exception for each one? 14:52:28 <ChrisPriceAB> dneary, the part - Where no standard is available provide API support references 14:53:16 <dneary> For example, CIMI, or ETSI NFV IFA would be relevant standards, but I don't think OpenStack is in compliance with either 14:53:20 <ChrisPriceAB> how many standards are there that are ignored? The point is that we should look to use standard interfaces rather than proprietary (even in open source they are proprietary) interfaces. 14:54:08 <ChrisPriceAB> Part of the point is to start that discussion also. Capture the conclusions and document the motivation fot his method of compliance/certification testing. 14:54:13 <dneary> I would say "not standardized" rather than "proprietary" but I take your point 14:55:26 <rprakash> Here is where C&C pointed out that since more than 50% of our work involves Openstack and if we compare or do gap analysis on VIM (OpenStack part) that itself will throw questions on differences and time to fix the gaps 14:55:33 <ChrisPriceAB> Yes and we may also test standards that are not adopted in the longer term. This is unavoidable, but as long as we stay up to date, relevant, and provide quality metrics it is all good. 14:57:01 <ChrisPriceAB> Neither C&C nor dovetail have it in scope to be doing time to fix gap analysis. We should validate agreed use capabilities, until we can do that well let's leave any additional scope out of discussion. 14:57:21 <rprakash> That is why just tesing APIs will miss the Applying Use cases that may not cover all APIs over different atleast have a OPNFV Platform as a Blackbox and get us moving to fix API gaps asa we move forward in Rle D 14:58:14 <ChrisPriceAB> Desist from trying to drive development activity in dovetail. Please attend those needs to the functest and yardstick team meetings. 14:58:15 <dneary> rprakash, I agree with ChrisPriceAB here - identifying and fixing API gaps (gaps to do what?) is out of scope for Dovetail 14:58:40 <dneary> If APIs are missing for a use-case, there should be a project around that use-case, identifying and implementing those APIs in the right place 14:59:33 <dneary> If Dovetail shows up areas where more alignment is needed in the platform, great! But characterizing what that alignment is, and doing the work, is not part of a conformance testing activity 14:59:42 <dneary> We test if things match what we say they should 14:59:48 <matthewli_> I am working in test projects, try to understand all those principles 14:59:58 <ChrisPriceAB> So can we agree to start a test case need, structure, requirements wiki to identify "how" we will test? 15:00:03 <dneary> And if we can't even have our own platforms match our conformance test, then our conformance test suite is broken 15:00:17 <dneary> ChrisPriceAB, I agree with that 15:00:17 <rprakash> OK so do we agree that Dovetail will only do use case testing and feed the missing API gaps to prjects to fix in OPNFV or upstream? 15:00:41 <dneary> rprakash, Let's consider ChrisPriceAB's proposal to agfree on test case requirements first please 15:01:09 <ChrisPriceAB> rprakash, I agree that dovetail should work accoridng to it's scope : https://wiki.opnfv.org/display/dovetail 15:01:27 <rprakash> #agree test case need, structure, requirements wiki to identify "how" we will test? 15:01:58 <rprakash> #agree refer link for scope link https://wiki.opnfv.org/display/dovetail 15:02:47 <ChrisPriceAB> What do those statements mean rprakash? 15:03:08 <dneary> #undo 15:03:08 <collabot> Removing item from minutes: <MeetBot.ircmeeting.items.Agreed object at 0x2847310> 15:03:29 <dneary> #link https://wiki.opnfv.org/display/dovetail - a reminder of Dovetail scope 15:04:06 <rprakash> Now for structure if we look at OPenstack they have TCUP, DefCore and RefStack, do we then propose on that line or use Yardstick, Functest and other framework? 15:04:24 <matthewli_> Not API test 15:04:55 <dneary> I have to go to another meeting 15:05:02 <dneary> Can we close on that positive note? 15:05:12 <rprakash> Excellent 15:05:16 <ChrisPriceAB> rprakash. Please, let's first define what we intend to produce. Then we can figure out the best tools to produce it. 15:06:18 <rprakash> #info we agree to follow on defining what we have on record here and close the meeting now and meet next week 15:06:31 <rprakash> #endmeeting