After tuning my system to develop an Android app, I noticed that every time ADB (Android Debug Bridge) daemon was started, the mouse cursor (and all graphics for that matter, as well as other I/O tasks like playing music) in my GNU/Linux system would start flickering, staggering, jumping, get slow, I'm not even sure what to call it. Besides that, the CPU usage was always around 2% to 7% (if seen from KSysGuard, lower using ps).
KSysGuard shows the ADB daemon consuming an average 4% CPU time
After monitoring the adb process and system I/O operations, I noticed that the problem was probably related to ADB polling the host USB devices periodically. That made me think of the disastrous xHCI controller from Fresco Logic, which I have in my computer for providing USB 3.0 connectivity, that had already given me problems before.

For now, what I do to avoid this issue when I need ADB is simply unloading the kernel module xhci_hcd:
sudo modprobe -r xhci_hcd
If I didn't want to use my USB 3.0 port I would just blacklist the module.

If you have this issue, post a comment or contact me so we can have a further chat and possibly get more insight.