Tag: tracking

Multi Object Camera Tracking

I wrote a ROS (Robot Operating System) package for 2-dimensional tracking in realtime an arbitrary number of moving objects using computer vision (opencv), kalman filtering, and data association. Data are saved as hdf5 files and analyzed with the python-pandas package. I have successfully used the system for tracking and analyzing trajectories of up to 10 flies for over 24 hours.

Codebase:  https://github.com/florisvb/multi_tracker

Multi-modal sensory integration in Mosquitoes (2015)

Mosquitoes find human hosts through the iteration of several independent behavioral modules, which are linked together through environmental interactions.
Mosquitoes find human hosts through the iteration of several independent behavioral modules, which are linked together through environmental interactions.

To find human hosts, mosquitoes must integrate sensory cues that are separated in space and time. To solve this challenge my collaborators Michael Dickinson, Jeff Riffell, and Adrienne Fairhall and I showed that mosquitoes respond to exhaled CO2 by exploring visual features they otherwise ignore. This guides them to potential hosts, where they use cues such as heat and humidity to locate a landing site.

Coauthored with: Jeff Riffell, Adrienne Fairhall, Michael Dickinson. Read more about our work in Current Biology.

 

 

Animation: The above animations show a collection of 200-500 mosquito trajectories, each aligned to the moment when they last passed through a CO2 plume. Only trajectories that approach either the room temperature (blue), or the 37° C (orange), object are shown. Note that the mosquito trajectories were recorded at different times, and superimposed for presentation purposes. Only 20 mosquitoes were released into the (1.5x.3x.3 m^3) wind tunnel at a time, and rarely were there more than a few flying simultaneously – in flight interactions were rare. Note how the mosquitoes spend more time near the warm object.

Neuromodulation of Flight Speed Regulation (2014)

Confocal image of a Drosophila brain and ventral nerve cord showing the GFP-labeled octopamine neurons (green), which we genetically silenced with Kir2.1. Red labels neuropil. Image taken by my colleague Marie Suver.
Confocal image of a Drosophila brain and ventral nerve cord showing the GFP-labeled octopamine neurons (green), which we genetically silenced with Kir2.1. Red labels neuropil. Image taken by my colleague Marie Suver.

Recent evidence suggests that flies’ sensitivity to large-field optic flow is increased by the release of octopamine during flight. This increase in gain presumably enhances visually mediated behaviors such as the active regulation of forward speed, a process that involves the comparison of a vision-based estimate of velocity with an internal set point. To determine where in the neural circuit this comparison is made, we selectively silenced the octopamine neurons in the fruit fly Drosophila, and examined the effect on vision-based velocity regulation in free-flying flies. We found that flies with inactivated octopamine neurons accelerated more slowly in response to visual motion than control flies, but maintained nearly the same baseline flight speed. Our results are parsimonious with a circuit architecture in which the internal control signal is injected into the visual motion pathway upstream of the interneuron network that estimates groundspeed.

Coauthored with Marie Suver, Michael Dickinson. Read more at the Journal of Experimental Biology.

Preferred visual motion set point is either modulated by changes in gain identical to those applied in the lobula plate tangential cell (LPTC) network (H2a), or the set point enters the visual sensory-motor cascade upstream of the LPTC network (H2b). (A) Block diagram showing the models under consideration. (B) Gain versus temporal frequency curve used for the low pass filter in the visual sensory system of our model. The data points are drawn from the temporal frequency tuning curve given in fig. 1D in Suver et al. (Suver et al., 2012), which summarizes the responses of electrophysiological recordings of vertical system (VS) cells in response to vertical motion. The original data were scaled such that the gain at a temporal frequency of 1 Hz is 1. The line shows a third-order polynomial fit. Note that this results in a transfer function defined in the linear temporal frequency domain, rather than the oscillatory temporal frequency domain. In order to implement this type of filter in our control model, we calculate the gain based on the linear temporal frequency of the stimulus. (C) Baseline subtracted membrane potential of VS cells in response to a downward 8 Hz visual motion stimulus during flight; data repeated, and magnified, from Suver et al. (Suver et al., 2012). The gray traces show the mean responses each of 19 individual flies, and the bold trace shows the group mean. (D) Model predictions compared with our results from Fig. 3E. The solid blue line shows the model prediction for the parental controls (gain=5.5) with the biomechanical saturation, whereas the dotted blue line shows the prediction without saturation. The solid red line shows the model prediction for the flies with inactivated octopamine neurons (gain=2.2). Note that the models H1, H2a and H2b all give identical acceleration responses. (E) Model predictions (color coded consistently with A) compared with mean velocity versus time responses for parental controls (left) and flies with inactivated octopamine neurons (right). The data traces are repeated from Fig. 3A. Note that H2 is a better fit.
Preferred visual motion set point is either modulated by changes in gain identical to those applied in the lobula plate tangential cell (LPTC) network (H2a), or the set point enters the visual sensory-motor cascade upstream of the LPTC network (H2b). (A) Block diagram showing the models under consideration. (B) Gain versus temporal frequency curve used for the low pass filter in the visual sensory system of our model. The data points are drawn from the temporal frequency tuning curve given in fig. 1D in Suver et al. (Suver et al., 2012), which summarizes the responses of electrophysiological recordings of vertical system (VS) cells in response to vertical motion. The original data were scaled such that the gain at a temporal frequency of 1 Hz is 1. The line shows a third-order polynomial fit. Note that this results in a transfer function defined in the linear temporal frequency domain, rather than the oscillatory temporal frequency domain. In order to implement this type of filter in our control model, we calculate the gain based on the linear temporal frequency of the stimulus. (C) Baseline subtracted membrane potential of VS cells in response to a downward 8 Hz visual motion stimulus during flight; data repeated, and magnified, from Suver et al. (Suver et al., 2012). The gray traces show the mean responses each of 19 individual flies, and the bold trace shows the group mean. (D) Model predictions compared with our results from Fig. 3E. The solid blue line shows the model prediction for the parental controls (gain=5.5) with the biomechanical saturation, whereas the dotted blue line shows the prediction without saturation. The solid red line shows the model prediction for the flies with inactivated octopamine neurons (gain=2.2). Note that the models H1, H2a and H2b all give identical acceleration responses.
(E) Model predictions (color coded consistently with A) compared with mean velocity versus time responses for parental controls (left) and flies with inactivated octopamine neurons (right). The data traces are repeated from Fig. 3A. Note that H2 is a better fit.

Odor plume tracking in Drosophila (2014)

A long exposure of a fruit fly illuminated by a shaft of light as the fly approaches a fermenting strawberry in a wind tunnel.
A long exposure of a fruit fly illuminated by a shaft of light as the fly approaches a fermenting strawberry in a wind tunnel.

Background: For a fruit fly, locating fermenting fruit where it can feed, find mates, and lay eggs is an essential and difficult task requiring the integration of olfactory and visual cues. Here, we develop an approach to correlate flies’ free-flight behavior with their olfactory experience under different wind and visual conditions, yielding new insight into plume tracking based on over 70 hr of data.

Results: To localize an odor source, flies exhibit three iterative, independent, reflex-driven behaviors, which remain constant through repeated encounters of the same stimulus: (1) 190 6 75 ms after encountering a plume, flies increase their flight speed and turn upwind, using visual cues to determine wind direction. Due to this substantial response delay, flies pass through the plume shortly after entering it. (2) 450 6 165 ms after losing the plume, flies initiate a series of vertical and horizontal casts, using visual cues to maintain a crosswind heading. (3) After sensing an attractive odor, flies exhibit an enhanced attraction to small visual features, which increases their probability of finding the plume’s source.

Conclusions: Due to plume structure and sensory-motor delays, a fly’s olfactory experience during foraging flights consists of short bursts of odor stimulation. As a con- sequence, delays in the onset of crosswind casting and the increased attraction to visual features are necessary behav- ioral components for efficiently locating an odor source. Our results provide a quantitative behavioral background for eluci- dating the neural basis of plume tracking using genetic and physiological approaches.

Coauthored with Michael Dickinson. Read more about my work in Current Biology.