14:02:22 <mackonstan> #startmeeting FD.io CSIT project meeting
14:02:22 <collabot`> Meeting started Wed Oct  2 14:02:22 2019 UTC.  The chair is mackonstan. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:02:22 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:02:22 <collabot`> The meeting name has been set to 'fd_io_csit_project_meeting'
14:02:28 <mackonstan> #chair
14:02:28 <collabot`> Current chairs: mackonstan
14:02:30 <tifrank> #info Tibor Frank
14:02:37 <vrpolak> #info Vratko Polak.
14:02:46 <jgelety> #info Jan Gelety
14:04:06 <mackonstan> #topic Agenda bashing
14:04:37 <mackonstan> #topic FD.io CSIT lab infrastructure
14:06:48 <mackonstan> #info Juraj: want to add two thunderX2 servers for vpp-device tests. And use existing thunderX2 for 2-node tests.
14:09:15 <mackonstan> #info Juraj: have question re power for new servers - will contact vexxhost team.
14:10:55 <mackonstan> #info CLX servers: Peter - CLX perf servers running, in all perf jobs including trending. Presented in trending pages as 2n-clx.
14:12:32 <mackonstan> #action Peter: implement PBF functionality (priority based frequency) based on Intel recommendation. Draft tech proposal of PBF usage to be run by csit-dev.
14:13:00 <mackonstan> #topic Inputs from LFN and FD.io projects
14:14:29 <mackonstan> #info VPP: check vpp meeting notes re vpp v19.08.2 status
14:14:43 <mackonstan> #info TSC: no update
14:14:55 <mackonstan> #topic Releases
14:16:03 <mackonstan> #info CSIT-1908 report updates: reconf tests graphs replaced by link pointing to CSIT-1908_1 report with corrected test results after reconf tests fixes got applied.
14:23:58 <mackonstan> #info CSIT-1908.1 report: Tibor - generated this morning CEST. Missing some runs for 3n-skx, 3n-hsw, 3n-tsh. Report data has been validated for presentation, but need to validate data before announcing.
14:25:30 <mackonstan> #info CSIT-2001: plan to be captured on wiki
14:25:38 <mackonstan> #link https://wiki.fd.io/view/CSIT/csit2001_plan
14:26:05 <mackonstan> #topic Operational status
14:26:40 <mackonstan> #info Jan: issue with vpp-device environment. Peter fixed. Peter to do TOI for future similar situations.
14:28:26 <mackonstan> #info Jan: trending shows failing multi-core tests, mostly 4c, some 2c. Suspecting Ole's patch touching stats handling by workers. Already interacting on slack channel #csit-dev.
14:34:25 <tifrank> #info Info from Vanessa:The Nexus maintenance was completed successfully. While performance appears to have improved, the hung jobs issue is not resolved. We modified the cronjobs on Monday. The next step is to move to the global-jjb lf-publish macro for logs. I'm working with Ed Kern to make sure the macro works within the nomad containers.
14:34:37 <mackonstan> #info Tibor: still observing some data upload and download failures - 19(1 failure), 22(1), 24(2), 26(1)-Sep, so more sporadic failures despite lots of activity (lots of uploads/downloads, many generations of report). Could be a different problem causing connectivity errors. Will paste this update into the helpdesk ticket.
14:35:18 <mackonstan> #info Tibor: no issues observed since Sunday.
14:36:40 <mackonstan> #info Ed: heads-up - another jenkins maintenance to apply changes listed by Vanessa to address the ongoing problem.
14:39:32 <mackonstan> #info Maciek: re Centos CI vpp-device - Thomas F. Herbert confirmed that he will have a limited bandwidth to support this. Will send a note to the list.
14:42:31 <mackonstan> #info Peter: Centos environment is missing keys. Ed to take it on, and see if it can be addressed in a similar fashion as done for vpp project.
14:44:22 <mackonstan> #info HoneyComb tests, Maciek: close the action re removing HoneyComb tests from CSIT repo, as HC project is dormant and lost sync with VPP.
14:44:55 <mackonstan> #topic VPP code performance
14:45:45 <mackonstan> #info trending - no new issues, apart from the multi-core listed by Jan earlier.
14:46:49 <mackonstan> #info Tibor: planning to add code to generate emails with anomaly information similar format as for failures, based on trending analytics
14:47:03 <mackonstan> #topic Developments
14:48:05 <mackonstan> #info VAT to PAPI: Jan - completed. One open item: scale tests.
14:48:20 <mackonstan> #info move to Python3: Jan - no update.
14:51:04 <mackonstan> #info vpp-api crc checks - Vratko: code updated to address vpp patch racing condition. Improved process description merged. Awaiting another vpp patch situation to have it fully verified in production.
14:52:34 <mackonstan> #info Vratko: next proposal for aligning csit and vpp master branches, driven by experience with vpp api crc checks. Removes need for vpp_stable and csit oper branches. See patch:
14:53:05 <snergster> #info current LF plan of record on timeout issue is to switch to new volume type on jenkins that was already done to nexus. that outage should be scheduled before eow.  If that doesnt get around the issue they want to take a look at changing the log pulling to either console log OR console timestamp log but not both.
14:53:30 <mackonstan> #info Vratko: working on a script for automated git bisecting performance regressions.
14:54:21 <vrpolak> #link https://gerrit.fd.io/r/c/csit/+/22354  <-- Draft document for future improvements of API process.
14:55:43 <mackonstan> #info Peter: vpp in unprivileged containers - making good progress. Issues with hugepages… vfio-pci handling is wip.
14:55:44 <vrpolak> #link https://gerrit.fd.io/r/c/csit/+/22261  <-- Work in progress for script that bisects to locate a regression.
14:55:54 <mackonstan> #topic Test environments
14:57:22 <mackonstan> #info re VIRL shutdown - Jan/Peter: only one test left running - dot1ad (already covered by performance test).
14:59:11 <mackonstan> #info re VIRL shutdown - Maciek: proposed course of action - review any other dependencies relying on VIRL env (nsh_sfc, hc, …), address those by for the last final time contacting the projects, and upon resolution, remove the not needed tests (VIRL mainly), and follow this by shutting down VIRL environment.
15:00:04 <mackonstan> #info re VIRL shutdown - proposal(s) for repurposing the VIRL servers.
15:00:40 <mackonstan> #info Peter: consider leaving one VIRL server for any remaining tasks that need VIRL.
15:01:33 <mackonstan> #endmeeting