08:00:17 <morgan_orange> #startmeeting Functest weekly meeting January 19th 08:00:17 <collabot`> Meeting started Tue Jan 19 08:00:17 2016 UTC. The chair is morgan_orange. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:00:17 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic. 08:00:17 <collabot`> The meeting name has been set to 'functest_weekly_meeting_january_19th' 08:00:25 <amaged__> #info Ahmed Maged 08:00:27 <viktor_nokia> #info Viktor Tikkanen 08:00:28 <juhak> #info Juha Kosonen 08:00:33 <morgan_orange> #info Morgan Richomme 08:00:40 <jose_lausuch> #info Jose Lausuch 08:00:48 <morgan_orange> hello everybody 08:01:14 <morgan_orange> #info agenda: https://wiki.opnfv.org/functest_meeting 08:01:26 <morgan_orange> #info mainly focusing on B release at Release -2 weeks 08:01:43 <morgan_orange> #topic action points follow-up 08:01:55 <morgan_orange> #link http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2016/opnfv-testperf.2016-01-12-08.00.html 08:02:15 <morgan_orange> #info AP 1: run test onos 08:02:28 <morgan_orange> May-meimei: lixiaoguang any update ? 08:02:49 <morgan_orange> #info AP 2: Jira reorganization for odl => done 08:03:23 <morgan_orange> #info AP 3 => complete onos section in Functest done => pending (May-meimei lixiaoguang) 08:03:31 <May-meimei> morgan_orange: hello 08:03:47 <morgan_orange> #info AP 4: prepare doctor env => done (and also promise and ovno) 08:04:07 <morgan_orange> #info AP5 jose_lausuch move rally to 0.2.0 => done 08:04:22 <morgan_orange> #info AP6 morgan_orange creation of next release page => not done, focus on B release 08:04:30 <morgan_orange> hello May-meimei 08:04:39 <May-meimei> morgan_orange: compass+onos test partly ok 08:04:40 <morgan_orange> did you succeed in running Onos test suite? 08:04:44 <morgan_orange> cool 08:04:55 <May-meimei> vping failed 08:05:07 <morgan_orange> I saw this morning that 18/18 passed tests on ODL => compass first installer getting this results 08:05:12 <jose_lausuch> May-meimei: can you do a manual vping test? 08:05:16 <May-meimei> because of metedata 08:05:23 <jose_lausuch> create 2 vms, ssh to one and ping the other one? manually? 08:05:30 <May-meimei> onos can't support that 08:05:48 <May-meimei> jose_lausuch: manual vping is ok 08:05:53 <jose_lausuch> we use "userdata" in vping, that was causing problems some time ago until we fixed it 08:06:02 <jose_lausuch> so, in compass we can't use userdata in nova? 08:06:06 <morgan_orange> it depends on the config, onos already supported vping 08:06:18 <jose_lausuch> onos ? 08:06:18 <morgan_orange> #topic B release situation 08:06:38 <May-meimei> morgan_orange: jose_lausuch: manual vping on compass+onos is ok 08:06:59 <morgan_orange> #info manual vping on compass+onos is ok but issue during last CI run 08:07:09 <May-meimei> jose_lausuch: morgan_orange: but compass+nosdn and compass+odl will be ok 08:07:44 <morgan_orange> #info new pages created to capture situation for B Release 08:07:53 <May-meimei> compass+odl have come other issues 08:08:06 <morgan_orange> #link https://wiki.opnfv.org/vping_brahmaputra_page 08:08:13 <morgan_orange> #link https://wiki.opnfv.org/tempest_brahmaputra_page 08:08:19 <morgan_orange> #link https://wiki.opnfv.org/vims_brahmaputra_page 08:08:30 <morgan_orange> may be we shall proceed installer by installer 08:08:35 <jose_lausuch> ok 08:08:43 <morgan_orange> #action amaged__ create odl page for brahmaputra 08:08:53 <viktor_nokia> BTW, I'm completely lost with available scenarios, do we have a complete list of them somewhere? 08:09:14 <morgan_orange> viktor_nokia: there 2 lists of scenario 08:09:25 <morgan_orange> 1 list = available scenario = jenkins 08:09:39 <morgan_orange> https://build.opnfv.org/ci/view/OPNFV%20Platform%20CI%20-%20Alternative%20View/ 08:09:45 <morgan_orange> 1 list ) candidate scenario 08:09:54 <morgan_orange> https://wiki.opnfv.org/brahmaputra_testing_page 08:10:12 <morgan_orange> this second list is a bit fuzzy because scenario are sometimes described several times 08:10:22 <Zhanghaoyu> hi ,morgan there are some testcase onosfw does not supported, could we pick them out when we run CI functest? 08:10:39 <jose_lausuch> Zhanghaoyu: what things? 08:10:45 <jose_lausuch> which testcases? 08:10:50 <viktor_nokia> morgan_orange: OK, thanks! 08:10:51 <morgan_orange> the wiki was used to identify which scenario were needed, once implemented (i.e. jenkins job) => they are removed from the wiki list 08:11:02 <morgan_orange> so basically our scenario = jenkins scenario 08:11:07 <morgan_orange> for B release 08:11:28 <morgan_orange> the convergence on the name of the scenario has been agreed only 1 week ago... 08:11:35 <Zhanghaoyu> metadata test case 08:11:42 <morgan_orange> teh format is os-[controller]-[feature]-[mode] 08:12:05 <morgan_orange> with controller = nosdn (pure neutron), odl_l2, odl_l3, onos, ocl 08:12:12 <morgan_orange> mode = ha, non ha 08:12:13 <jose_lausuch> we parse [controller] in functest and select the proper test to run 08:12:16 <jose_lausuch> same for feature 08:12:22 <morgan_orange> et feature = ovs (dpdk), kvm,... 08:13:07 <morgan_orange> in CI we can put constraints on any test case. With these constraints we build dynamically the list of tests runnable on this scenario 08:13:09 <David_Orange> not "non ha" but "noha" 08:13:46 <jose_lausuch> David_Orange: correct, thanks :) 08:13:51 <morgan_orange> let(s me info some info 08:14:00 <jose_lausuch> the constraints for each test case is defined in config_functest.yaml 08:14:01 <morgan_orange> #info question on the scenario 08:14:21 <morgan_orange> #info available scenario = jenkins 08:14:33 <morgan_orange> #link https://build.opnfv.org/ci/view/OPNFV%20Platform%20CI%20-%20Alternative%20View/ 08:14:42 <morgan_orange> #info candidate scenario (not yet in jenkins) = wiki 08:14:48 <morgan_orange> #link https://wiki.opnfv.org/brahmaputra_testing_page 08:14:57 <morgan_orange> #info for B release we must support available scenario 08:15:18 <morgan_orange> #info moving from candidate to available is a question between feature & installer projects 08:15:33 <morgan_orange> #info question on conditions to run the tests 08:15:47 <morgan_orange> #info mechanism in place in functest, we can put constraint on each testcase 08:16:10 <morgan_orange> #info list of runnable testcase defined based on the comparison between CI info (scenario, controller) and contraints defined statically in functest 08:16:36 <morgan_orange> #link https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml 08:16:45 <morgan_orange> #info at the end of this file you may see the constriants 08:17:12 <morgan_orange> but the approach is to associate constraints to a testcase not to exclude 08:17:32 <morgan_orange> so we may say for instance that vPing must work only in apex, fuel, joid and not compass 08:17:32 <Zhanghaoyu> jose_lausuch: is there any config file for us to config which testcase can not be runed in CI job? 08:17:59 <jose_lausuch> Zhanghaoyu: you just got an answer from morgan_orange :) 08:18:21 <morgan_orange> Zhanghaoyu: but in your case you would like to exclude a configuration right? 08:19:10 <morgan_orange> if we consider vping => no constraint so we systematically will try it towards any fresh installation 08:19:21 <morgan_orange> the system of constraint is based on regex 08:20:06 <morgan_orange> e.g. promise must be tested on fuel or joid 08:20:15 <morgan_orange> not sure we can exclude a config 08:20:17 <morgan_orange> to be checked 08:20:23 <morgan_orange> back to installer 08:20:27 <morgan_orange> status on apex 08:20:30 <morgan_orange> #info status on apex 08:20:47 <morgan_orange> #link https://build.opnfv.org/ci/view/functest/job/functest-apex-opnfv-jump-1-daily-master/lastSuccessfulBuild/consoleText 08:21:24 <morgan_orange> issue with vPing as well 08:21:32 <jose_lausuch> morgan_orange: https://build.opnfv.org/ci/view/functest/job/functest-apex-opnfv-jump-1-daily-master/20/consoleFull 08:21:41 <jose_lausuch> lastSuccessfulBuild might be different every time 08:21:43 <jose_lausuch> take the id 08:21:55 <jose_lausuch> ya, vping failed 08:21:57 <Zhanghaoyu> morgan_orange: https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml in this file right? actually some testcase we dont support now 08:22:02 <jose_lausuch> and we have another small problem 08:22:04 <jose_lausuch> Zhanghaoyu: yes 08:22:17 <jose_lausuch> juhak: viktor_nokia: something strange happens with tempest results 08:22:31 <morgan_orange> OK Zhanghaoyu let me indicate that offline 08:23:08 <jose_lausuch> ====== 08:23:08 <jose_lausuch> Totals 08:23:08 <jose_lausuch> ====== 08:23:08 <jose_lausuch> Ran: 97 tests in 160.0000 sec. 08:23:08 <jose_lausuch> - Passed: 89 08:23:08 <jose_lausuch> - Skipped: 0 08:23:09 <jose_lausuch> - Expected Fail: 0 08:23:09 <jose_lausuch> - Unexpected Success: 0 08:23:10 <jose_lausuch> - Failed: 8 08:23:11 <jose_lausuch> Sum of execute time for each test: 314.3107 sec. 08:23:23 <jose_lausuch> but the, if you do a rally verify list 08:23:25 <jose_lausuch> you get 08:23:30 <jose_lausuch> Results: {'timestart': '2016-01-1823:24:28.399601', 'duration': 169, 'tests': 113, 'failures': 24} 08:23:34 <jose_lausuch> different numbers 08:23:53 <Zhanghaoyu> morgan_orange: im a little affraid about whether these testcase take some negative effect on our ONOSFW release 08:24:45 <Zhanghaoyu> jose_lausuch: if we pick them out it don't have some problems about B release? 08:24:53 <morgan_orange> Zhanghaoyu: why? apex has also some trouve and inidcate for instance that vIMS case will not be supported 08:25:43 <Zhanghaoyu> :) just a little worried about it, thank you for your response~ 08:25:54 <morgan_orange> BTW on apex/onos Zhanghaoyu May-meimei lixiaoguang => ImportError: No module named pxssh 08:26:04 <morgan_orange> I think that is the error I saw during the week end 08:26:11 <morgan_orange> seems a library is missing 08:26:49 <viktor_nokia> jose_lausuch: It looks like "setUpClass ... ... FAILED" cases are counted as 1 test/failure in Totals but actually more than one test case is skipped (=failed) because of setUp failure 08:26:53 <morgan_orange> requirements file must be updated 08:27:17 <morgan_orange> pexpect must be added to go further with onos test run from CI 08:27:42 <viktor_nokia> jose_lausuch: this is just a theory, I'm not sure 08:27:52 <morgan_orange> #action May-meimei add pexpect lib in https://git.opnfv.org/cgit/functest/tree/docker/requirements.pip 08:28:05 <jose_lausuch> viktor_nokia: ok.. its a bit confusing 08:28:38 <morgan_orange> rally is running (we will make a specific topic for rally after installer review) 08:28:54 <jose_lausuch> viktor_nokia: even the number of tests is different 08:29:01 <jose_lausuch> and the time 08:29:02 <jose_lausuch> Ran: 112 tests in 93.0000 sec. 08:29:09 <jose_lausuch> duration': 105, 'tests': 120 08:29:38 <morgan_orange> regarding Tempest viktor_nokia created the page https://wiki.opnfv.org/tempest_brahmaputra_page 08:29:44 <viktor_nokia> jose_lausuch: this is indeed confusing since the number of tests differs in ~80...200 range 08:29:55 <May-meimei> morgan_orange: I am opening at the requirements.pip, If have some question,I will ask you later 08:30:04 <morgan_orange> May-meimei: ok 08:30:49 <morgan_orange> and apex/onos => not started due to lib issue, apex/odl => 15 passed (3 failed) 08:31:01 <jose_lausuch> I was writing the same:) 08:31:12 <morgan_orange> on compass last night it was 18/18 passed so the bug we used to have in Arno seems to be solved 08:31:19 <jose_lausuch> nice 08:31:35 <viktor_nokia> how about success criteria for test suites? 08:31:41 <morgan_orange> amaged__: raghavendrachari pbandzi could you have a look and help us to troubleshoot current odl suite on the different installers 08:31:53 <morgan_orange> viktor_nokia: in theory or in practice... 08:32:03 <jose_lausuch> :) 08:32:05 <pbandzi> hi morgan_orange 08:32:08 <viktor_nokia> both :) 08:32:12 <raghavendrachari> ok 08:32:59 <morgan_orange> "In theory, theory and practice are the same. In practice, they are not." Einstein 08:33:10 <viktor_nokia> I mean how we define status in those brahmaputra testing pages? 08:33:20 <jose_lausuch> morgan_orange: lol 08:33:43 <morgan_orange> Initially I would have liked to use the dashboard and avoid wiki pages that are unmaintanable.. 08:34:12 <morgan_orange> for tempest we define our own list 08:34:22 <morgan_orange> so we may expect a 0 failed 08:34:25 <Zhanghaoyu> morgan_orange: hi morgan,onos do not support vPing,how to modify this yaml? 08:34:33 <Zhanghaoyu> test-dependencies: 08:34:33 <Zhanghaoyu> doctor: 08:34:33 <Zhanghaoyu> installer: '(apex)|(fuel)' 08:34:33 <Zhanghaoyu> functest: 08:34:33 <Zhanghaoyu> vims: 08:34:33 <Zhanghaoyu> vping: 08:34:34 <Zhanghaoyu> tempest: 08:34:35 <Zhanghaoyu> rally: 08:34:35 <Zhanghaoyu> odl: 08:34:36 <Zhanghaoyu> scenario: 'odl' 08:34:36 <Zhanghaoyu> onos: 08:34:36 <Zhanghaoyu> scenario: 'onos' 08:34:50 <morgan_orange> but it is impossible, the introduction of odl for instance triggers lots of network errors 08:35:21 <morgan_orange> so the criteria is OK we we have a reasonable number of tests failes (10%) assuming that we can explain the reason why it fails... 08:35:47 <viktor_nokia> OK 08:36:18 <Zhanghaoyu> morgan_orange: BTW on apex/onos Zhanghaoyu May-meimei lixiaoguang => ImportError: No module named pxssh ----i'm fixing it 08:36:30 <morgan_orange> Zhanghaoyu: the proper way to exclude vPing from onos is to add a constraint on scenario so something like scenario: nosdn|odl|ocl 08:36:42 <morgan_orange> should exclude any scenario with onos 08:36:53 <viktor_nokia> May be we can use colors (green for 100% yellow for 90-100% and red for others) 08:36:56 <morgan_orange> but vPing is very basic 08:37:08 <morgan_orange> could you detail offline the reason why it should be excluded? 08:37:56 <morgan_orange> viktor_nokia: the difficulty is the complexity of combinations...I think it will be hard to reach 90% on all the scenario...but we can try 08:38:14 <morgan_orange> for vPing is basic => ok or failed 08:38:36 <morgan_orange> for vIMS => we do not check the results of the tests but the capability to deploy the orchestrator, deploy the vIMS and run the tests 08:38:57 <morgan_orange> for odl => as compass succeeded to get a 18/18 it should be the target for any odl scenario... 08:39:06 <jose_lausuch> morgan_orange: good news: we got 18 tests total, 18 passed, 0 failed in Fuel+ODL ! 08:39:11 <morgan_orange> for onos ups to ONOSFW to decide its criteria 08:39:24 <morgan_orange> for tempest => 90% OK 08:39:26 <jose_lausuch> https://build.opnfv.org/ci/view/functest/job/functest-fuel-opnfv-jump-2-daily-brahmaputra/22/consoleFull 08:39:30 <morgan_orange> for rally => 90% OK 08:39:39 <morgan_orange> and error failures must be documented 08:39:44 <morgan_orange> is is ok for everybody? 08:39:59 <jose_lausuch> sure 08:40:02 <viktor_nokia> OK 08:40:20 <May-meimei> morgan_orange, jose_lausuch: so we don't have to pass all the testcase? 08:40:30 <jose_lausuch> but for tempest we have missleading result numbers... we have to figure it out 08:40:50 <viktor_nokia> I will check them 08:40:51 <jose_lausuch> May-meimei: I think if we require that constraint, release B won't happen :D 08:40:54 <morgan_orange> #info success criteria: vPing 100% OK, odl 100% ok, Tempest 90% OK, Rally 90% OK, vIMS 100%OK (deploy+run tests do not care of test result) 08:41:30 <morgan_orange> we move to the compass installer? 08:41:33 <Zhanghaoyu> morgan_orange: CNlucius will explain the reason why onos not support vPing testcase 08:42:10 <carey_xu> onos does not support metadata yet, so vm can't get userdata 08:42:12 <May-meimei> morgan_orange, jose_lausuch: sorry I left for a while, about compass+onos, vping test is fail, I think jose_lausuch understand the reason, yes? 08:42:14 <morgan_orange> Zhanghaoyu: ok thanks, if needed we may make some adaptations (as spawning 2 VM in a private network and performing a ping between tehm seems a reaonable testcase....) 08:42:16 <jose_lausuch> Zhanghaoyu: CNlucius is it user data problem ? 08:42:37 <morgan_orange> #info status compass installer 08:42:51 <CNlucius> yes 08:43:07 <morgan_orange> #info issue vPing towards compass/onos 08:43:08 <jose_lausuch> CNlucius: we can think of another way of doing that maybe 08:43:23 <jose_lausuch> instead of passing userdata, we pass a script and SCP to the vm to ping the other vm 08:43:27 <jose_lausuch> is that ok? 08:43:35 <morgan_orange> I think it makes sens to adapt vPing if needed ot keep the testcase as it is a basic Hellow world 08:43:49 <jose_lausuch> and we get rid of user data... 08:43:53 <carey_xu> morgan_orange: can we force config_drive? 08:43:57 <jose_lausuch> the problem is 08:44:07 <jose_lausuch> we need floating IPs if we want to SCP 08:44:21 <carey_xu> so cloud-init can get userdata from config_drive? 08:44:24 <jose_lausuch> carey_xu: then, are we sure config_drive will work always? 08:44:37 <CNlucius> That would be best 08:45:08 <morgan_orange> OK let's discuss that offline 08:45:08 <carey_xu> jose_lausuch: need verify 08:45:12 <morgan_orange> there is 15 minutes left 08:45:22 <morgan_orange> I would like to speak about rally and documentations 08:45:30 <jose_lausuch> carey_xu: CNlucius: let's discuss it after the meeting 08:45:32 <morgan_orange> and feature projects.. 08:45:49 <morgan_orange> just info the lats main results for the installer 08:46:15 <morgan_orange> #info status fuel 08:46:23 <morgan_orange> #info on track 08:46:28 <morgan_orange> #info status joid 08:46:53 <morgan_orange> #info still lots of errors, but David_Orange told me that ODL has been upgraded yesterday... 08:47:07 <morgan_orange> vPing is also in timeout on joid 08:47:27 <morgan_orange> odl fully failed on joid/odl 08:47:37 <morgan_orange> (only installer with this status) 08:47:43 <jose_lausuch> it could be because of a different port 08:47:49 <morgan_orange> #info vPing is also in timeout on joid 08:47:55 <morgan_orange> #info odl fully failed on joid/odl 08:48:01 <jose_lausuch> maybe its not 8181 08:48:17 <morgan_orange> jose_lausuch: yep but we asked several times the confirmation of the 8181 port, maybe an @ issue 08:48:21 <morgan_orange> troubleshooting needed 08:48:32 <morgan_orange> #topic rally 08:48:46 <morgan_orange> #info rally-cert scenario ready 08:49:13 <morgan_orange> #info they may need some config adaptations but it is possible now to switch from our traditional way of running rally to this mode 08:49:32 <morgan_orange> #info are we ok to move assuming that we did not perform manual tests (not access on all the labs) 08:49:41 <morgan_orange> #info number of tests almost the same 08:50:22 <jose_lausuch> juhak: please review this 08:50:22 <jose_lausuch> https://gerrit.opnfv.org/gerrit/#/c/6897/ 08:50:29 <morgan_orange> I would suggest to migrate ..then troubleshoot on the different installers 08:50:29 <jose_lausuch> to avoid printing all the output 08:50:51 <morgan_orange> jose_lausuch: what do you think? 08:50:55 <jose_lausuch> I tried it manually and it works, it prints only the stdout 08:51:05 <juhak> I think it would be good to do some manual testing with rally-cert first 08:51:32 <jose_lausuch> morgan_orange: +1 with rally cert, but also +1 to test first manually 08:51:38 <jose_lausuch> I can do that on fuel lf-pod2 08:51:40 <morgan_orange> on which lab can we do that => LF POD1 (apex) and LF POD2 (fuel) 08:51:50 <jose_lausuch> I have a time window for testing 08:51:50 <morgan_orange> May-meimei: could you try a manual run on compass using this script 08:52:36 <morgan_orange> #info test rally-cert on LF POD1, LF POD2, Huawei-US and Intel POD5 =>synchro with juhak, jose_lausuch, morgan_orange if OK migration to rally-cert 08:52:49 <morgan_orange> target test today... 08:52:56 <morgan_orange> ok? 08:53:04 <juhak> ok 08:53:07 <jose_lausuch> ok 08:53:19 <jose_lausuch> looks like a busy day 08:53:21 <May-meimei> ok! 08:53:30 <morgan_orange> #topic doc 08:53:36 * jose_lausuch likes challenges 08:53:40 <morgan_orange> #info developer doc initiated (under review) 08:53:53 <morgan_orange> #info user and installation/config doc to be patched 08:54:21 <morgan_orange> question on onos, do you have your own doc, or do you want to be integrated in these documents (we host odl test doc) 08:54:50 <morgan_orange> if you want to be described in the doc, you must patch the current documents there are sections for ONOS.. 08:55:19 <morgan_orange> #action morgan_orange May-meimei jose_lausuch test rally-cert and do a synch with juhak for scenario migration in CI 08:56:00 <morgan_orange> #action lixiaoguang May-meimei update user guide to include onos doc (if not own onos documentation / otherwise just indicate a pointer) 08:56:08 <jose_lausuch> ok 08:56:15 <morgan_orange> any question on the documents that are supposed to be freeze also today... 08:56:16 <May-meimei> morgan_orange: ok 08:56:38 <jose_lausuch> #action carey_xu CNlucius check userdata problem and propose a solution to adapt vPing to make it work 08:56:47 <jose_lausuch> also me 08:57:22 <morgan_orange> there is also a B plan in progress on test dashboard, the demo last thursday was a bit disapointing and clearly not up to date 08:57:25 <carey_xu> jose_lausuch: OK 08:57:35 <CNlucius> ok 08:57:37 <Zhanghaoyu> morgan_orange: thank you so much, and we are handling docs 08:57:40 <jose_lausuch> morgan_orange: what is plan b? 08:57:43 <morgan_orange> we are considering using our data results (from the DB) through an existing framework 08:58:40 <morgan_orange> pbandzi: frankbrockners have deployed a ELK in a VM where I moved the mongo DB, I am able to consume the APi to generate through a cron job data ready for graphs 08:58:52 <morgan_orange> I have to see ghow to integrate them in graphana or .. 08:59:05 <jose_lausuch> ah ok, that one looked very good 08:59:07 <morgan_orange> it is not critical compared to the rest of our activities 08:59:23 <jose_lausuch> I think if we consider that dashboard looks more "pro", we should use that one 08:59:30 <Zhanghaoyu> morgan_orange: im new to this meeting, so whats the schedule of this meeting in recently, i want to join this. 08:59:33 <morgan_orange> but as the test projects made the effort to centralize their results and even provide post processing algo, it will be a pity not to use them 09:00:29 <morgan_orange> Zhanghaoyu: the meeting is the test group weekly meeting, time is not convenient for Asia (Thursday 3PM UTC) but we have also a slot for Asia (Wedesnday 8 AM UTC) 09:00:54 <morgan_orange> unfortunately last time nobody attended so I sent a mail to ask whether we should stop this slot 09:01:11 <morgan_orange> this meeting was created to share information betwwen all the test projects 09:01:30 <morgan_orange> ok we are one minute late 09:01:35 <morgan_orange> thanks for joining 09:01:41 <morgan_orange> good luck for this new very long day 09:01:42 <Zhanghaoyu> morgan_orange: you are a good guy~ this meeting helps me alot 09:01:42 <jose_lausuch> thanks morgan_orange 09:01:51 <jose_lausuch> everyone has something to do :) 09:02:02 <Zhanghaoyu> thank you so much 09:02:10 <morgan_orange> Zhanghaoyu: Jose wrote it in the Functest in depth presentation...we are good guys :) 09:02:21 <jose_lausuch> we are nice guys :) 09:02:25 <morgan_orange> if you have question keep on this chan 09:02:41 <Zhanghaoyu> :) 09:02:42 <morgan_orange> and we are looking for musicians to jam during the summit.. 09:02:48 <morgan_orange> #endmeeting