13:56:44 <mackonstan> #startmeeting FD.io CSIT Project Meeting 13:56:44 <collab-meetbot`> Meeting started Wed Jul 21 13:56:44 2021 UTC and is due to finish in 60 minutes. The chair is mackonstan. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:56:44 <collab-meetbot`> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:56:44 <collab-meetbot`> The meeting name has been set to 'fd_io_csit_project_meeting' 13:58:25 <pmikus> #info Peter Mikus 14:01:15 <mackonstan> #info mackonstan 14:01:17 <mackonstan> #chair 14:01:17 <collab-meetbot`> Current chairs: mackonstan 14:01:57 <vrpolak> #info Vratko Polak. 14:02:06 <dwallacelf> #info Dave Wallace 14:02:53 <mackonstan> #topic Agenda bashing 14:03:43 <mackonstan> #topic Inputs from projects - VPP 14:04:07 <mackonstan> #topic Inputs from projects - TSC 14:04:09 <jlinkes> #info Juraj Linkeš 14:05:42 <mackonstan> #info Dave, Peter Mikus nominated as ci-management committer, all committers voted in favour, TSC approval this week. 14:06:09 <mackonstan> #topic New tests 14:06:51 <fan_zhang> #info Fan Zhang 14:08:08 <mackonstan> #info Fan, re flow "offload" patch functional test, waiting for vpp_device testbed/framework update. 14:09:20 <mackonstan> #info Fan, re flow "offload" patch performance test, testbeds/framework ready, will ask Yulong to go ahead and create a patch for that. 14:11:42 <mackonstan> #info Maciek, re TRex support for DPDK CVL, will send a summary note re status, will need help to verify any remaining gaps. 14:13:41 <mackonstan> #info Juraj, re IPsec SPD flow cache patches, 3 patches open, found one issues, working on a fix. Vratko wants to run perf tests to see any collateral impact of those patches. 14:14:14 <mackonstan> #topic Releases 14:15:27 <mackonstan> #info CSIT-2110, will follow VPP schedule 14:15:29 <mackonstan> #link https://wiki.fd.io/view/Projects/vpp/Release_Plans/Release_Plan_21.10 14:17:30 <mackonstan> #info CSIT-2106, Tibor, still testing (delayed by jenkins issues, talked about later), 2n-zn2 1 cvg job, af_xdp driver tests (late arrival to job_specs). 14:19:54 <mackonstan> #info CSIT-2106, Tibor, methodology sections under review, ipsec async patch, uni-dir crypto, af_xdp content to be added (Peter, Maciek). 14:21:51 <mackonstan> #info CSIT-2106, Peter, MLX issue fixed for good. 14:23:22 <mackonstan> #info CSIT-2106, Vratko, re IPsec async broken test, still open; Fan, vpp patch may need to be re-worked. 14:25:01 <mackonstan> #info CSIT-2106, high latency issue reported by TRex, Vratko, one very clear symptom: latency running high to 5msec, some NICs affected some are not, same for testbeds. 14:25:51 <mackonstan> #info Vratko, issue appeared when testbeds got upgraded to Ubuntu 20.04 and TRex v2.88. Unclear what is the root cause. One similar symptom reported in public domain for TRex. 14:30:50 <vrpolak> #link https://github.com/cisco-system-traffic-generator/trex-core/issues/508#issuecomment-662363667 <-- Known TRex issue, comment about impact on latency measurements, but for different firmware and TRex versions. 14:31:43 <mackonstan> #info Peter, also tried TRex v2.90, also observing high latency. 14:34:20 <mackonstan> #info Peter, another suspicion is PCIe bus management, checking ... 14:37:35 <mackonstan> #info CSIT-2101, vpp artefacts ready, will test once CSIT-2106 is complete, after latency issue is fixed, and one iterative job per testbed is run with latency fixed. Then CSIT-2101.1 maintenance rls tests will get scheduled. 14:37:53 <mackonstan> #topic Physical Infrastructure 14:38:36 <mackonstan> #info Peter, going ahead with 3n-hsw decommissioning, tickets open. Waiting for Vexxhost engineer on-site engineer allocation. 14:38:51 <mackonstan> #topic Virtual Infrastructure, CI/CD 14:39:31 <mackonstan> Dave, Maciek, escalated Jenkins outage issue 14:39:44 <mackonstan> #info Dave, Maciek, escalated Jenkins outage issue 14:44:14 <mackonstan> #info Dave, trigger: last Wednesday Jenkins VM and Jenkins ingress got moved to different location(s), some other changes on Saturday; symptoms: as of Saturday, very high latency between jenkins ingress and nomad; resolution: migrated jenkins ingress somewhere else; Peter cleaned vpp_device jobs, Dave cleared the vpp job queue. Now back to normal 14:44:14 <mackonstan> operation. 14:46:22 <mackonstan> #info Dave, jenkins maintenance not needed 14:47:11 <mackonstan> #info Dave, Vanessa applied Jenkins JCasC changes during the outage. 14:47:20 <mackonstan> #info Peter, Dave, Maciek 14:47:53 <mackonstan> #info Peter, Dave, Maciek, changes of this nature MUST BE communicated over the slack channel. 14:51:04 <mackonstan> #info Dave, Maciek: will request post mortem readout from Vexxhost/LFIT team. 14:51:54 <mackonstan> #topic Trending 14:52:24 <mackonstan> #info Vratko, CSIT l2, ip4, ip6 scale tests are failing due to VPP removing support for VAT APIs used. 14:53:58 <mackonstan> #info Vratko, solution is still WIP. 14:54:59 <mackonstan> #endmeeting