15:31:12 <dfarrell07> #startmeeting s3p
15:31:12 <odl_meetbot> Meeting started Fri Jul 29 15:31:12 2016 UTC.  The chair is dfarrell07. Information about MeetBot at http://ci.openstack.org/meetbot.html.
15:31:12 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:31:12 <odl_meetbot> The meeting name has been set to 's3p'
15:31:40 <dfarrell07> #info Scale/perf weekly meeting, maybe a bit of clustering
15:32:29 <dfarrell07> #info Marcus is on honeymoon, enjoy! :)
15:32:33 <dfarrell07> #info Jamo is on PTO, enjoy! :)
15:32:40 <dfarrell07> #info Nikos is on PTO, enjoy! :)
15:32:53 <dfarrell07> #info dfarrell07 has been taking detailed look at NetVirt CSIT, Docker.py CSIT lib
15:33:00 <dfarrell07> #info Docker.py lib seems to have much of the logic we need
15:33:06 <dfarrell07> #info Looks like we can use Dave's socketplane/docker-ovs Docker image
15:33:11 <dfarrell07> #link https://github.com/socketplane/docker-ovs Best OVS Docker image (to scale in NetVirt test) dfarrell07 has found so far
15:33:18 <dfarrell07> #info Dave's OVS Docker image has OVS up to 2.4. Does that work? Need newer?
15:33:24 <dfarrell07> #info Docker.py lib doesn't seem to be used in any test, dfarrell07 to work on that next
15:33:37 <dfarrell07> #info Luis completed a bunch of clustering lib work this week, OpenFlow next
15:33:49 <dfarrell07> #info Al Morton of OPNFV CPerf/IETF (and others?) gave good feedback to Luis about OF tests/metrics, Luis to respond and continue discussion
15:33:58 <dfarrell07> #info NetVirt CSIT has a lot of jobs and a lot of failures, dfarrell07 is still trying to figure out which to build from, will talk to Jamo about this when he returns from PTO
15:34:03 <dfarrell07> #info GBP also has Docker libs but they are very minimal, not reusable, not what we want
15:34:16 <dfarrell07> #info LuisGomez is still looking for people to help with OpenFlow tests
15:34:37 <dfarrell07> #info Kumar will be helping Luis with OF tests. Thanks!
15:34:42 <dfarrell07> #undo
15:34:42 <odl_meetbot> Removing item from minutes: <MeetBot.ircmeeting.items.Info object at 0x23db350>
15:34:53 <dfarrell07> #info Kumar Rishabh (on call) will be helping Luis with OF tests. Thanks!
15:35:45 <dfarrell07> #info Current way of plotting perf results is result/run, which doesn't work well if doing a single run but changing a var like switch scale (vs time for example)
15:36:11 <dfarrell07> #info LuisGomez will be on PTO week of 8/8
15:37:55 <dfarrell07> #info Eric + Dileep: Dileep has been doing some profiling work, interested in adding that to monitoring work Kumar is working on
15:38:26 <dfarrell07> #info LuisGomez has recent additions to CSIT libs that do JVM monitoring, easy via Robot and ODL logs server
15:39:08 <dfarrell07> #info LuisGomez tried this monitoring framework with a CBench perf test very recently, didn't see a perf degradation, LuisGomez thought it would be heavy but it doesn't seem to be
15:39:28 <dfarrell07> #info LuisGomez will send out link to wiki with samples/image of monitoring tool
15:40:10 <dfarrell07> #info Dileep and Marcus have been talking about profiling and integrating that with tests
15:41:23 <dfarrell07> #info Dileep talks about it not being possible to combine data from multiple JVMs, like when running in a cluster
15:41:50 <dfarrell07> #info Dileep is looking at using yourkit, collecting and automating, but kinda a mess
15:42:05 <dfarrell07> #info Found another tool (missed name) that is built to run in multiple JVMs
15:42:30 <dfarrell07> #info This tool can combine JVM outputs to find non-intuitive issues
15:44:06 <dfarrell07> #info Another option is Karaf Decanter (which icbts worked on, I think)
15:44:43 <dfarrell07> #info LuisGomez reports that framework he has can work with Decanter, anything with JMX, first via Decanter then to DB then to plotting logic, also supports clustering
15:45:26 <dfarrell07> #info Keyword for this monitoring detects if in cluster or not, and automagically makes the correct number of graphs
15:47:35 <dfarrell07> #info Dileep gives overview of various JVM profiling tools he has investigated, tools to supplement JMX via Decanter
15:48:01 <dfarrell07> #info Use of flame graphs specifically seems very nice, Nikos has also pointed these out as a very good goal
15:49:10 <dfarrell07> #info Dileep gives demo of flame graphs, show stacks, where time is spent, very helpful
15:49:40 <dfarrell07> #info LuisGomez loves flame graph, would love to add, don't have today, wanted to start with simple things and add better profiling info like this
15:51:47 <dfarrell07> #info LuisGomez asks Dileep how far along this work is, if it will be ready for automation before whitepaper testing
15:52:51 <dfarrell07> #info If we don't have time to get this into automation, can at least do this profiling in Intel env
15:53:13 <dfarrell07> #info LuisGomez and dfarrell07 would *love* to see this added to monitoring framework upstream so can be used in CI/many jobs
15:54:32 <dfarrell07> #info First step is to get profiling tools installed in RelEng, then get them running in right JVMs, LuisGomez can point Dileep and past patches that do similar things for Decanter
15:55:43 <dfarrell07> #info Discussion about keeping graph (now) vs logging to results in DB and creating many graphs via dashboard (future)
15:56:29 <dfarrell07> #info Marcus has also been doing work to send data to external DB via Robot, and there's an intern and an Intel person working on dashboards, so may want to align
15:57:39 <dfarrell07> #info Int/Test people are pretty damn busy with release atm, but a few Intel people may be able to get this done fairly quickly, good examples/help to follow
15:58:06 <odp-gerritbot> A H proposed a change to integration/distribution: Adding eman features  https://git.opendaylight.org/gerrit/42792
15:59:02 <dfarrell07> #info Discussion about using Grafana, what TSDR is currently doing/not doing. LuisGomez thinks TSDR stuff is just for ODL DS, not generic info, like profiling results.
16:00:08 * icbts notices ByteMan symbol shinning in the night sky - to the ByteCave I go.
16:00:43 <icbts> We have some nice updates coming down the line to Decanter - i’ll be outlining them at the ODL SUmmit
16:01:15 <dfarrell07> #info icbts will be giving talk at ODL Summit about cool new Decanter features, we should attend
16:02:18 <dfarrell07> #info More discussion about where data goes and gets plotted
16:03:13 <dfarrell07> #info LuisGomez suggest Dileep go look at work Marcus is doing to put work in external DB, start with that path since this doesn't have to be done super-soon
16:04:04 <dfarrell07> #action LuisGomez to send intro email to right people, pointers to right patches, to Dileep about sending data to external DB
16:06:24 <dfarrell07> #info Dileep's email address is dileep.ranganathan@intel.com
16:08:13 <odp-gerritbot> A H proposed a change to integration/test: Add basic test suite for eman  https://git.opendaylight.org/gerrit/42793
16:09:12 <icbts> Decanter, Cave, Karaf-boot, Cellar, etc :)
16:09:38 <icbts> http://sched.co/7RBf
16:13:16 <dfarrell07> #link http://sched.co/7RBf Link to icbts's ODL Summit Karaf++ talk we should all attend
16:13:34 <dfarrell07> #info dfarrell07 gives NetVirt scale test updates, see #infos above
16:15:59 <dfarrell07> #info Sanjib gives updates about OF testing, data collection, stats collection scripts/logic
16:16:25 <dfarrell07> #info afredette knows about a newer Docker OVS image than the one dfarrell07 described above, will send pointer
16:17:29 <dfarrell07> #info Discussion about clustering deploying scripts
16:17:46 <afredette> #info docker images: https://lists.opendaylight.org/pipermail/netvirt-dev/2016-July/000700.html
16:18:19 <dfarrell07> #undo
16:18:19 <odl_meetbot> Removing item from minutes: <MeetBot.ircmeeting.items.Info object at 0x23404d0>
16:18:20 <dfarrell07> #link https://lists.opendaylight.org/pipermail/netvirt-dev/2016-July/000700.html Docker OVS image, new and recommended, OVS 2.5
16:18:26 <dfarrell07> afredette: thanks :)
16:19:54 <dfarrell07> #info LuisGomez will give TWS in 2 weeks about clustering lib
16:20:14 <dfarrell07> #info Idea is to make it very easy for projects to write their on CSIT with clustering, lib extracted/provided by Int/Test
16:20:42 <dfarrell07> #info Can also use this clustering lib in S3P tests, will start with single, see stable, then do in cluster
16:22:34 <dfarrell07> #info Discussion about normal isolation in clustering tests, which things run in which VMs
16:25:12 <dfarrell07> #info dfarrell07 fixed the meeting info for this meeting on GCal
16:25:40 <dfarrell07> #info More discussion about future of clustering tests, work, timelines, plans
16:26:24 <dfarrell07> #info LuisGomez points out that cluster just isn't very stable atm, so really need projects to fix that before can do too much more, Int/Test people need to keep testing/sending bugs
16:28:06 <dfarrell07> #info dfarrell07 reminds all that Jamo raised a bunch of bugs, especially about clustering, after ONS this year. Projects seem to have dropped the ball on them, we need to go back and push for fixes.
16:28:10 <dfarrell07> #endmeeting