#opendaylight-integration: s3p

Meeting summary

1. 1. Scale/perf weekly meeting, maybe a bit of clustering (dfarrell07, 15:31:40)
   2. Marcus is on honeymoon, enjoy! :) (dfarrell07, 15:32:29)
   3. Jamo is on PTO, enjoy! :) (dfarrell07, 15:32:33)
   4. Nikos is on PTO, enjoy! :) (dfarrell07, 15:32:40)
   5. dfarrell07 has been taking detailed look at NetVirt CSIT, Docker.py CSIT lib (dfarrell07, 15:32:53)
   6. Docker.py lib seems to have much of the logic we need (dfarrell07, 15:33:00)
   7. Looks like we can use Dave's socketplane/docker-ovs Docker image (dfarrell07, 15:33:06)
   8. https://github.com/socketplane/docker-ovs Best OVS Docker image (to scale in NetVirt test) dfarrell07 has found so far (dfarrell07, 15:33:11)
   9. Dave's OVS Docker image has OVS up to 2.4. Does that work? Need newer? (dfarrell07, 15:33:18)
   10. Docker.py lib doesn't seem to be used in any test, dfarrell07 to work on that next (dfarrell07, 15:33:24)
   11. Luis completed a bunch of clustering lib work this week, OpenFlow next (dfarrell07, 15:33:37)
   12. Al Morton of OPNFV CPerf/IETF (and others?) gave good feedback to Luis about OF tests/metrics, Luis to respond and continue discussion (dfarrell07, 15:33:49)
   13. NetVirt CSIT has a lot of jobs and a lot of failures, dfarrell07 is still trying to figure out which to build from, will talk to Jamo about this when he returns from PTO (dfarrell07, 15:33:58)
   14. GBP also has Docker libs but they are very minimal, not reusable, not what we want (dfarrell07, 15:34:03)
   15. LuisGomez is still looking for people to help with OpenFlow tests (dfarrell07, 15:34:16)
   16. Kumar Rishabh (on call) will be helping Luis with OF tests. Thanks! (dfarrell07, 15:34:53)
   17. Current way of plotting perf results is result/run, which doesn't work well if doing a single run but changing a var like switch scale (vs time for example) (dfarrell07, 15:35:45)
   18. LuisGomez will be on PTO week of 8/8 (dfarrell07, 15:36:11)
   19. Eric + Dileep: Dileep has been doing some profiling work, interested in adding that to monitoring work Kumar is working on (dfarrell07, 15:37:55)
   20. LuisGomez has recent additions to CSIT libs that do JVM monitoring, easy via Robot and ODL logs server (dfarrell07, 15:38:26)
   21. LuisGomez tried this monitoring framework with a CBench perf test very recently, didn't see a perf degradation, LuisGomez thought it would be heavy but it doesn't seem to be (dfarrell07, 15:39:08)
   22. LuisGomez will send out link to wiki with samples/image of monitoring tool (dfarrell07, 15:39:28)
   23. Dileep and Marcus have been talking about profiling and integrating that with tests (dfarrell07, 15:40:10)
   24. Dileep talks about it not being possible to combine data from multiple JVMs, like when running in a cluster (dfarrell07, 15:41:23)
   25. Dileep is looking at using yourkit, collecting and automating, but kinda a mess (dfarrell07, 15:41:50)
   26. Found another tool (missed name) that is built to run in multiple JVMs (dfarrell07, 15:42:05)
   27. This tool can combine JVM outputs to find non-intuitive issues (dfarrell07, 15:42:30)
   28. Another option is Karaf Decanter (which icbts worked on, I think) (dfarrell07, 15:44:06)
   29. LuisGomez reports that framework he has can work with Decanter, anything with JMX, first via Decanter then to DB then to plotting logic, also supports clustering (dfarrell07, 15:44:43)
   30. Keyword for this monitoring detects if in cluster or not, and automagically makes the correct number of graphs (dfarrell07, 15:45:26)
   31. Dileep gives overview of various JVM profiling tools he has investigated, tools to supplement JMX via Decanter (dfarrell07, 15:47:35)
   32. Use of flame graphs specifically seems very nice, Nikos has also pointed these out as a very good goal (dfarrell07, 15:48:01)
   33. Dileep gives demo of flame graphs, show stacks, where time is spent, very helpful (dfarrell07, 15:49:10)
   34. LuisGomez loves flame graph, would love to add, don't have today, wanted to start with simple things and add better profiling info like this (dfarrell07, 15:49:40)
   35. LuisGomez asks Dileep how far along this work is, if it will be ready for automation before whitepaper testing (dfarrell07, 15:51:47)
   36. If we don't have time to get this into automation, can at least do this profiling in Intel env (dfarrell07, 15:52:51)
   37. LuisGomez and dfarrell07 would *love* to see this added to monitoring framework upstream so can be used in CI/many jobs (dfarrell07, 15:53:13)
   38. First step is to get profiling tools installed in RelEng, then get them running in right JVMs, LuisGomez can point Dileep and past patches that do similar things for Decanter (dfarrell07, 15:54:32)
   39. Discussion about keeping graph (now) vs logging to results in DB and creating many graphs via dashboard (future) (dfarrell07, 15:55:43)
   40. Marcus has also been doing work to send data to external DB via Robot, and there's an intern and an Intel person working on dashboards, so may want to align (dfarrell07, 15:56:29)
   41. Int/Test people are pretty damn busy with release atm, but a few Intel people may be able to get this done fairly quickly, good examples/help to follow (dfarrell07, 15:57:39)
   42. Discussion about using Grafana, what TSDR is currently doing/not doing. LuisGomez thinks TSDR stuff is just for ODL DS, not generic info, like profiling results. (dfarrell07, 15:59:02)
   43. icbts will be giving talk at ODL Summit about cool new Decanter features, we should attend (dfarrell07, 16:01:15)
   44. More discussion about where data goes and gets plotted (dfarrell07, 16:02:18)
   45. LuisGomez suggest Dileep go look at work Marcus is doing to put work in external DB, start with that path since this doesn't have to be done super-soon (dfarrell07, 16:03:13)
   46. ACTION: LuisGomez to send intro email to right people, pointers to right patches, to Dileep about sending data to external DB (dfarrell07, 16:04:04)
   47. Dileep's email address is dileep.ranganathan@intel.com (dfarrell07, 16:06:24)
   48. http://sched.co/7RBf (icbts, 16:09:38)
   49. http://sched.co/7RBf Link to icbts's ODL Summit Karaf++ talk we should all attend (dfarrell07, 16:13:16)
   50. dfarrell07 gives NetVirt scale test updates, see #infos above (dfarrell07, 16:13:34)
   51. Sanjib gives updates about OF testing, data collection, stats collection scripts/logic (dfarrell07, 16:15:59)
   52. afredette knows about a newer Docker OVS image than the one dfarrell07 described above, will send pointer (dfarrell07, 16:16:25)
   53. Discussion about clustering deploying scripts (dfarrell07, 16:17:29)
   54. https://lists.opendaylight.org/pipermail/netvirt-dev/2016-July/000700.html Docker OVS image, new and recommended, OVS 2.5 (dfarrell07, 16:18:20)
   55. LuisGomez will give TWS in 2 weeks about clustering lib (dfarrell07, 16:19:54)
   56. Idea is to make it very easy for projects to write their on CSIT with clustering, lib extracted/provided by Int/Test (dfarrell07, 16:20:14)
   57. Can also use this clustering lib in S3P tests, will start with single, see stable, then do in cluster (dfarrell07, 16:20:42)
   58. Discussion about normal isolation in clustering tests, which things run in which VMs (dfarrell07, 16:22:34)
   59. dfarrell07 fixed the meeting info for this meeting on GCal (dfarrell07, 16:25:12)
   60. More discussion about future of clustering tests, work, timelines, plans (dfarrell07, 16:25:40)
   61. LuisGomez points out that cluster just isn't very stable atm, so really need projects to fix that before can do too much more, Int/Test people need to keep testing/sending bugs (dfarrell07, 16:26:24)
   62. dfarrell07 reminds all that Jamo raised a bunch of bugs, especially about clustering, after ONS this year. Projects seem to have dropped the ball on them, we need to go back and push for fixes. (dfarrell07, 16:28:06)

Action items

1. LuisGomez to send intro email to right people, pointers to right patches, to Dileep about sending data to external DB

People present (lines said)

1. dfarrell07 (67)
2. odl_meetbot (5)
3. icbts (4)
4. odp-gerritbot (2)
5. afredette (1)